fortran-lang / registry Goto Github PK

I think it would easier to just have a "Create namespace" button on the dashboard instead of having a dedicated menu entry for it.

In the registry server, we are following a token based Authentication, in which if the token is valid and belongs to a particular user in the database, we allow the user to access the API atleast.

But, it has it's downsides as well. For example, if a person has the valid token, they can send in request from any client and if the data is correct then API will perform it's task accordingly.

We can discuss the better approaches to perform authentication in this issue.

cc: @henilp105 @minhqdao @perazz

The login/sign up/forgot password form doesn't have a loading indicator, so once you hit, e.g. the Log In button, nothing happens until the login process is over. There should be a feedback so the user knows that the login process has been triggered. You could add a loading indicator inside the button that replaces the text while the process is ongoing.

I have considered the following format for Mongo database :

1. fpmregistry as the Database.
2. It would have 3 collections named users, packages,namespaces.
3. suggested format for users collection
   

{
    "name": "username",
    "email": "email",
    "password": "password hash",
    "role":["user","maintainer","admin"],
    "createdAt":"pythondatetime object",
    "lastLogin": "pythondatetime object",
    "lastLogout": "pythondatetime object",
    "authorOf": ["_id of package1","_id of package2"], 
    "maintainerOf": ["_id of package1", "_id of package2"],
    "pendingRequest":["_id of package1", "_id of package2"],
    "sessionId": "uuid",
}

4.suggested format for packages collection

{
    "name": "package name",
    "namespace":"_id of namespace",
    "isDeprecated": "boolean",
    "tarball":"aws s3 object name",
    "version": "version",
    "license": "license name",
    "createdAt": "pythondatetime object",
    "updatedAt":"pythondatetime object",
    "author": "_id object",
    "maintainers": ["_id of user1","_id of user2"],
    "copyright": "copyright statement",
    "description": "package description",
    "tags":["t1","t2"],
    "dependencies": ["_id of package1","_id of package2"],
}

5.suggested format for namespaces collection

{
    "namespace": "namespace name",
    "createdAt":"pythondatetime object",
    "createdBy": "_id of user",
    "description": "namespace description",
    "tags":["t1","t2"],
    "authors": ["id of a1", "_id of a2"],
    "packages":["_id of package1","_id of package2"],
}

I would like to request all the Community members to review and add suggestions on this format of database.

Thanks and Regards,
Henil

CC @awvwgk @fortran-lang/admins @fortran-lang/fpm

The frontend does not support providing ratings, it only supports the display of package ratings.

CC @arteevraina @certik

Let's define the api for uploading packages. How do you want a JSON request for uploading a package to look like? @arteevraina @henilp105

Currently the tarball includes all dependencies (recursively). That will not scale. Instead, each tarball should just contain sources of the package itself, and all dependencies should be specified via fpm.toml, and the fpm package manager will correctly download those separately.

I think the README should provide all the relevant links to reach our backend services:

• The frontend of the package manager.
• The base url of the backend.
• The api documentations.

Please also keep it up-to-date in case there are any changes.

While we already have a repository at fortran-lang/fpm-registry it seems best to start from scratch to build the registry backend. This means that the packages listed there have to be resubmitted once the registry backend goes online, but this should be a minor issue as there are only few registered packages and almost all maintainers are still active in the Fortran community.

We can however use the already registered packages in the other repository as example packages for testing our registry backend.

Low priority issue.

I'm playing with the fpm publish command a bit and I've found that fitpack cannot be uploaded:

federico@Federicos-MBP fitpack % fpmx publish --token=XXXXXX
 Uploading package ...
{"code":400,"message":"Invalid license identifier"}

The license in fpm.toml is BSD-3. Any hints @arteevraina @henilp105 ?

The following APIs :

1. Get namespace admins.
2. Get namespace maintainers.
3. Get package maintainers.

These APIs are public. It would be essential to add necessary security checks for the same in the backend as well.

Ever since we moved to https://render.com to host our backend, I feel like the performance went down quite a bit. Especially cold starts (when there was no incoming request for a while, probably 30-60 min), a simple GET request to obtain package data might take several seconds. Maybe we can do something about it.

I have considered the following API Routes for backend so that they can be directly used by the website and the local registry/fpm :

1. /packages/?query=<name or tag or description keywords>
   This endpoint would send a GET request to API backend to fetch all the packages with name in package name, tag in package, keywords in description of package and return them in a json format.

2. /packages/<package_name>
   This endpoint would send a GET request to API backend to fetch the package tarball from aws s3 object and to return the tarball of package.

3. /auth/login
   This endpoint would send a GET/POST request to API backend to verify if the user exists and has exactly same password and email if yes, updates the loginAt field in users collection, generates a unique uuid sets it users collection and returns a uuid as a cookie.
   

4. /auth/logout
   This endpoint would send a GET/POST request to API backend to logout if the user exists and has exactly same uuid if yes, updates the logoutAt field in users collection, removes uuid users collection and resets cookie with uuid and returns json.
   

5. /auth/signup
   This endpoint would send a GET/POST request to API backend to verify if the user exists and has exactly same email if yes, shows errors else , creates a new user in users collection with the details,and adds the loginAt field in users collection, generates a unique uuid sets it users collection and returns a uuid as a cookie.

6. /users/<username>/
   This endpoint would send a GET request to API backend to verify by uuid and then to return all the packages uploaded by the user and their maintainer in json and also return is_admin, is_maintainer ( for the web interface, just to show the options in the menu to navigate to another API to edit packages/users which would be authenticated. ) .
   if user is admin then he would be shown options to search/show all the users to delete them and to add a new user with different levels to access and to search/edit/delete the packages, make an new user a maintainer. if the user is maintainer then he would be add/edit/delete different versions of the packages he would be maintaining.

7. /packages
   This endpoint would send a POST request to API backend to verify by uuid and then to add a new package or a version to the collection and then also request the maintainer to join as a maintainer if approved on maintainer side , returns a json with status code.

8. /packages/package-id
   This endpoint would send a PUT request to API backend to verify by uuid and then to update a package details to collection and then also request the maintainer to approve, returns a json with status code.

9. /users/<username>/<package_id>/delete
   This endpoint would send a GET/POST request to API backend to verify by uuid and then to delete a package from collection and returns a json with status code.

10. /users/<username>/maintainer/
    This endpoint would send a GET/POST request to API backend to only for web interface and shows the loggedin user the requests from a different packages with author names if the user approves them he would be added as a maintainer for that package, returns a json with status code.

11. /auth/forgot-password/
    This endpoint would send a GET/POST request to API backend to only for web interface and checks if an email exists if it exists then it would send the reset link with a new generated uuid by mail to user, returns a json with status code.

12. /auth/reset-password/<uuid>
    This endpoint would send a GET/POST request to API backend to only for web interface and checks if uuid exists if it exists then it would ask to enter a new password and change in the users collection and remove the uuid and resets cookie with uuid , returns a json with status code.

if the user is admin them direct access would be granted to delete/update/upload any package/user.
I would like to request all the Community members to review and add suggestions on this format of APIs. Please feel free to add here if I have missed any other important aspect.

Thanks and Regards,
Henil

CC @awvwgk @fortran-lang/admins @fortran-lang/fpm

It would be great to support both tar and zip formats and it should be easy to add that on the backend, too. Just allow the user to upload and download a zipball equivalent to a tarball.

In the project, there have been discussions on namespace maintainers & package maintainers.
Let's say if we have multiple packages under a namespace and if the admin adds a new maintainer to the namespace then the newly added maintainer will have access to all the packages that are listed under that namespace.

But, on the other hand, if we only have maintainers of the package. So, this security issue won't be there as that newly added maintainer will have access to that only package under that namespace and other packages will be not accessible to the maintainer.

On the other hand, we can allow user to add maintainers to the packages as well as to the namespaces. The logic will be similar, only the entity package will be changed with the namespace.

So, if an admin adds a user as a maintainer to the namespace, the user will be now the maintainer of all the packages under that namespace and if admin adds a user as a maintainer to the package, the user will be the maintainer of the package only.

Also, we can allow to remove maintainers from the packages easily. For removing maintainers from the namespaces, we can have multiple solutions. So, if a user wants to remove a maintainer from namespace, the maintainer can be directly removed from all the packages under that namespace or they can ask the backend to keep the user as the maintainer of the packages and only remove the maintainer from the namespace.

It would be really helpful if can gather a wider community feedback to better implement this feature. @fortran-lang/fpm @minhqdao @perazz @henilp105

we would add this set of env variables for the execution of github action with the variable name : TEST_ENV_FILE for the successful execution of the test cases.

MONGO_DB_NAME=testregistry
MONGO_URI=mongodb://localhost:27017/
MONGO_USER_NAME=Admin
MONGO_PASSWORD=Admin
SALT=MYSALT
HOST=localhost
[email protected]
RESET_PASSWORD=reset
SUDO_PASSWORD=fortran
IS_CI=true

The CI execution can be tested at my fork.

CC: @awvwgk @certik

Hi, I am trying to search the fpm package form home page, but the page is just keep loading.

here is the image after inspect:

Aim of this project is to develop a web server that will allow the uploading and downloading of packages. Packages will have names and version numbers, and be associated with one or more maintainers. Users must have an account and be logged in to upload packages. A package maintainer will be able to assign other maintainers and remove a package from the registry - either a specific version or the package in its entirety. Only a package maintainer will be allowed to upload new versions of an existing package.

T1: Initial Prototype of the web server

2 months (M1-M2)
The initial prototype will be capable of creating user accounts and allowing logged-in users to upload packages. It will allow package maintainers to remove a specific version of a package, or the package in its entirety. It will include sufficient tools for moderation and administration. I.e. administrators will be able to assign new maintainers to packages, delete user accounts, remove packages, etc. Finally, users will be able to download packages via a well-defined API without authentication.

T2: Upload Restrictions and Checks

2 months (M3-M4; requires T1)
One important check and restriction for a package on upload will be to ensure that all modules provided by a package are prefixed with the name of the package. This will ensure that module name conflicts do not arise between different packages. It is estimated that it will take 1 month to implement this check.
Another important aspect will be to ensure that the package can be built as a dependency.
As other checks and restrictions are determined, we will implement them as time allows.

T3: API for Query and Mirror

1 month (M5; requires T1)
To improve efficiency and reliability for users, we will implement an API allowing tools to easily query and mirror the repository. This will involve obtaining a list of all packages and versions, and bulk downloading of packages.

T4: fpm Support

1 month (M6)
Enable fpm to support uploading packages, querying the registry for available packages and versions, and fetching packages and their dependencies.

Deliverables

• D1: Prototype of a web server.
  • Period: M2.
  • Mean of verification: Web page available as frontend
• D2: Implementation of restrictions and checks.
  • Period: M4.
  • Mean of verification: See package violating checks fail to upload
• D3: API for query and mirror.
  • Period: M4.
  • Mean of verification: Access to API
• D4: Prototype version supporting fpm.
  • Period: M5:
  • Mean of verification: Use fpm to interact with the registry
• D5: Prototype supporting restrictions, checks, query and mirror, and fpm.
  • Period: M6.
  • Mean of verification: Use fpm and/or web frontend to interact with the registry. Implementation available on GitHub

Compare the package model of an uploaded package against the model in the registry to ensure no malicious code was added. Also for module naming conventions.

Details:

• The fpm builds a local package into a source tarball, and exports the model to JSON. It packages the JSON file with the source tarball.
• fpm uploads the source tarball with the JSON to registry
• The registry backend checks the JSON model against a model generated in the backend from fpm.toml from the uploaded source tarball

This check is not for security, but rather for consistency of the uploaded packages, it checks:

• checks for module naming conventions
• checking that fpm model can be built (so fpm.toml and all source files are consistent)

A separate feature to consider for a separate milestone is:

• Actually doing fpm build, that is, building the package using a Fortran compiler. This is effectively a CI. That's a huge milestone on its own.

As final packages are supposed to be immutable (except for some metadata maybe), I think we should have sth like a staging state (a simple field in the package like is_staging or is_final might suffice, or sth more sophisticated like a publication_state : "staging") where the package is uploaded and fully functioning but might be altered, swapped, or even removed. It acts as a beta phase before it becomes fully permanent within the database. I can just otherwise imagine that someone uploads a package which is buggy or has a typo and that'll stay in the database forever. 😣

On the fpm side, we could warn a user if a package is used that is still in staging mode. We can also warn the uploader that a package will become fully permanent and immutable if the he/she changes the package to "final".

The reset password link that I receive and when I make a GET request to this url. I get a Not Found error.

For example :

    Dear arteevraina,

    We received a request to reset your password. To reset your password, please copy paste the link below in a new browser window:

    localhost/account/reset-password/7bbb451c7b514c59a8ce00fb5fa586e8

    Thank you,
    The Fortran-lang Team

Package Updatability :
Should we consider keeping the packages mutable ? ( if yes , I would only like to keep the parameters isDeprecated only as mutable as keeping other parameters mutable might raise a lot of vulnerabilities and threat scenarios) , But for some Exceptional cases we would like to give admins power to change all the parameters of package.

Backend Hosting considerations :
I have currently hosted on vercel, it works very well and heroku both can work as great alternatives to AWS ( our earlier consideration) to reduce costs and also help in easy maintainability.

Mongodb Hosting considerations :
I have currently hosted on MongoDB Atlas it works very well and also gives good interface to the database (for easy access than a docker container) , But it adds slight latency to the APIs requests. But It would be easier to manage and would help minimise the downtime that we might have in our mongodb containers.

I would like to request all the Community members to review and add suggestions.

Thanks and Regards,
Henil

CC @awvwgk @fortran-lang/admins @fortran-lang/fpm @arteevraina @perazz @minhqdao

There's user feedback missing after hitting the logout button. First, nothing happens, then the nav bar all of a sudden changes. There should be some form of user feedback after starting a request.

Schema for package search API:

parameters:

• name: query
  description: package search string
  required: true
  type: string

• name: page
  description: page number of the query , 10 documents per page
  required: true
  type: string

• name: sorted_by
  description: package sort parameter can be name, author, createdat ,updatedAt. (case insensitive)
  required: false
  type: string

• name: sort
  description: sort by ascending (asc) or descending (desc)
  required: false
  type: string

Curl Request would be similar to:
curl --request GET \ --url 'http://localhost:5000/packages?query=fpm&page=1&sorted_by=name&sort=asc'

Thanks and Regards,
Henil

CC @arteevraina @minhqdao @perazz

A "dry run" for uploading a package means that the uploaded package goes through every validation step but in the end isn't stored in the database. It is a "simulated run". However, it is necessary that the dry run is performed via the backend to make sure that all the requirements of the backend for uploading the package are met.

We could optionally include verification of the token. If the token is provided, it will be validated. If it is missing, only the package will be validated without the token.

Instead of a success message like "Upload successful.", a message like "Dry run successful." should be returned. In case a validation step failes, the same error message would be returned as if it was an actual upload attempt.

I have Hosted all the User CRUD API and search , upload package , user main home page here : https://henilp105.vercel.app/ , Please feel free to do testing and I would like to request all the Community members to review and add suggestions on this User CRUD APIs PR #3 . Please feel free to add here if I have missed any other important aspect. This frontend is a template and would request to use CURL to ping the APIs.

Thanks and Regards,
Henil

CC @awvwgk @certik @fortran-lang/admins @fortran-lang/fpm @arteevraina @minhqdao @perazz @milancurcic

I have considered the following options for the addition of dependencies of packages to its document in packages collection :

1. Read the following from toml file.
2. User enters the packages in csv manner.
3. Add a dropdown to add the packages, by searching and selecting packages.

In my opinion it is generally better to add a dropdown or directly get them from the toml file of the packages.
I would like to request all the Community members to review and add suggestions.

Thanks and Regards,
Henil

CC @awvwgk @fortran-lang/admins @fortran-lang/fpm @arteevraina @perazz @minhqdao

The transfer account functionality allows sudo admins to transfer accounts of inactive/non-maintained packages/namespaces to a different user , to develop this functionality, I have considered the following methods to implement this functionality, this functionality would only be allowed to be executed to sudo admins.

we could directly allow the sudo admin to transfer the account , allow the sudo admin to change username, email, and send a password reset request to new email and all the packages and the namespace access would be directly transferred to the new username and old account would not be deleted and we could show the account has been transferred and redirect them to the new account ( by this method only when the packages are seen on the /users/<username> of the frontend would change , else every where it would be exactly similar to the original.

I would like to request all the Community members to review and add suggestions/discussions of this functionality.

Thanks and Regards,
Henil Panchal

CC @certik @awvwgk @arteevraina @minhqdao @perazz @milancurcic @everythingfunctional

I have considered the following Frontend Frameworks for making the registry webpage:

1. Flask
   Easy to integrate, Cheap won't require a separate hosting, easy to maintain resides with flask templates.
2. React / nodejs
   Easy to integrate, , difficult to set up and would require separate hosting.
3. Flutter
   Easy but it is not scalable, more inclined towards mobile than web.

In the following considerations I would like to consider to use flask for making the frontend interface, for the design theme I would like to make it similar to the theme of the webpage which was designed by me and @awvwgk ( fortran + pydata custom theme).

I would like to request all the Community members to review and add suggestions on frontend frameworks . Please feel free to add here.

Thanks and Regards,
Henil

CC @awvwgk @fortran-lang/admins @minhqdao @perazz @arteevraina

Since I and @henilp105 have already started working on implementing the APIs in the backend. @minhqdao suggested that we should document the APIs very well so that it will be easier to develop asynchronously without having the need to ping me and @henilp105 in the backend.

I am thinking we can maintain a markdown file inside the project in which we can list endpoints and describe how it receives the request and send json response and other things related to the API.

But, if anyone has any easier / better approach that will work please feel free to suggest in the comments.

@henilp105 @minhqdao @perazz @fortran-lang/fpm

@arteevraina @henilp105 @perazz

I've built and deployed a simple backend so I could implement the parsing on the fpm side. As that is basically done, there are a few things to consider in terms of the api so everything works flawlessly:

• The endpoint to download a specific version should be a GET request: <base_url>/packages/namespace/name/1.2.3.
• The endpoint to download the latest version of package is a POST request: <base_url>/packages/namespace/name.
• With the POST request, i'm sending a json with a list of cached versions via curl's -d parameter (it should be the same with wget's --post-data):

json = {
  "cached_versions": ["0.1.0", "0.2.0"]
}

curl <url> -d json -o <output>

• cached_versions is an array of strings and required, but it could be empty.
• Furthermore, I currently expect a json response from both requests that both contain the following fields: code, version and tar:

response = {
  ...
  "code": 200,
  "version": "0.1.0",
  "tar": "https://...",
  ...
}

• An error is currently thrown in fpm if any of the three fields aren't present.
• A side note on the document scheme because we just talked about it: Even if the version should be contained in the manifest, too, I think it is normal to have such essential data as a field in the document (yes, redundancy is ok) because that speeds up queries (mostly within the backend). Imagine you want to find the latest version of a package, you just want to compare the version fields and not parse every single manifest file to find out its version.
• By executing another curl or wget command to download from the url (tar's value), I'm currently expecting a tar.gz file.
• If I unpack and unzip the file with tar -zxf <file> -C <destination>, I expect all the contents of a package including src directory and manifest to be on the same level as <destination>. So the manifest should end up here <destination>/fpm.toml, not here <destination>/example_package/fpm.toml after unpacking.
• If you'd like to change any of these keys (e.g., if you prefer url over tar or status over code) just let me know and I can change or implement it in fpm.

• D1.1: Implemented metrics for uploaded packages
  • Ratings and stars
    • Backend
    • Frontend @arteevraina
  • Downloading of packages using fpm from the registry
  • Dependency resolution for downloading the package
• D1.2: Added possibility to report packages
  • Backend
  • Frontend for reporting and generating malicious reports
    • create a PR and merge #66
• D1.3: Provided functionality for security related incidents
  • Backend
  • Frontend: #69
    • seeing reported packages in the admin console
    • Seeing other incidents
• D1.4: Improved upload restrictions in backend
  • Backend infrastructure
  • CI #71
  • #65 #70

After these milestones are delivered, we need to get alpha tester to test it and we will fix bugs.

Bonus:

• Merge this PR: fortran-lang/fpm#879