GithubHelp home page GithubHelp logo

webysther / packagist-mirror Goto Github PK

View Code? Open in Web Editor NEW
189.0 12.0 68.0 3.52 MB

📦✂️📋📦 Create a mirror of packagist.org metadata for use locally with composer

Home Page: https://packagist.org/mirrors

License: MIT License

PHP 93.81% Dockerfile 2.45% Shell 3.74%
mirror packagist crawler php composer-packages composer packagist-mirror

packagist-mirror's Introduction

📦 Packagist Mirror

docker pulls Minimum PHP Version Packagist Codecov Quality Score Mentioned in Awesome composer

❤️ Recommended by packagist.org ❤️

This mirror is for Composer 1; Composer 2 is very fast on its own. We will update to support the version 2 for those need solve the slow internet access or availability problem with the main repository.

A mirror for packagist.org that regularly caches packages from one or more main mirrors to add to a distributed package repository.

Mirror creation

If you're using PHP Composer, commands like create-project, require, update, remove are often used. When those commands are executed, Composer will download information from the packages that are needed also from dependent packages. The number of json files downloaded depends on the complexity of the packages which are going to be used. The further you are from the location of the packagist.org server, the more time is needed to download json files. By using a mirror, it will save you time when downloading json because the server location is closer.

⚙️ How it works?

This project aims to create a local mirror with ease, allowing greater availability for companies/countries that want to use composer without depending on the infrastructure of third parties. It is also possible to create a public mirror to reduce the load on the main repository and better distribute requests around the world, helping make the packagist ecosystem faster as a whole!

When creating a mirror, you add a list of other mirrors to use for initial sync, which pulls all packages to your local machine. After the mirror is created and synced, the next runs will only pull updates. If any mirror fails to deliver a metadata file, the client will fallback to its configured main mirror, whether that be packagist.org or otherwise. If the client encounters an installation problem or loses connection to a mirror, it can return from where it stopped running.

Mirror creation

🌎 Packagist public metadata mirrors observatory around the world

🛫 Amazing data mirrors used to download repositories metadata built using this recommended repository or another:

Lists are ordered by country and sync frequency.

Location Mirror Maintainer Github Sync Since
Brazil packagist.com.br Webysther main Continuously Q3'17
China php.cnpkg.org Eagle Wu fork Every minute Q3'18
China packagist.mirrors.sjtug.sjtu.edu.cn Shanghai Jiao Tong University fork Every hour Q2'19
Czech Republic packagist.hostuj.to HOSTUJ TO fork Every 5 minutes 🆕Q1'20
Finland packagist.fi Niko Granö fork Continuously 🆕Q2'20
France packagist.fr Baptiste Pillot fork Every minute 🆕Q4'20
Germany packagist.hesse.im Benjamin Hesse fork Every minute 🆕Q3'20
Germany composer.mg100.net Alex Gummenscheimer fork Every minute 🆕Q1'21
India packagist.in Varun Sridharan fork Every minute Q2'19
India packagist.vrkansagara.in Vallabh Kansagara fork Every 5 minutes Q4'19
Indonesia packagist.phpindonesia.id Indra Gunawan fork Every 30 seconds Q3'18
Indonesia packagist.ianmustafa.com Ian Mustafa fork Every 30 seconds Q3'19
Indonesia packagist.telkomuniversity.ac.id Telkom University fork Every 5 minutes 🆕Q1'20
Japan packagist.dev.studio-umi.jp Studio Umi fork Every minute 🆕Q1'20
Russia packagist.org.ru Konstantin Tarasov fork Every 15 minutes Q3'22
South Africa packagist.co.za SolidWorx fork Every 5 minutes Q3'18
South Korea packagist.kr PackagistKR fork Every minute Q3'18
Thailand packagist.mycools.in.th Jarak Kritkiattisak fork Every 5 minutes Q4'19
USA packagist-mirror.wmcloud.org Wikimedia fork Every 5 minutes Q3'18
Taiwan packagist.tw Peter fork Every 5 minutes 🆕Q2'20
Vietnam packagist.ondinh.net Long Nguyen main Every 5 minutes 🆕Q3'20

⚠️ Not based on this source code:

Location Mirror Maintainer Github Sync Since
China mirrors.aliyun.com Aliyun Every 5 minutes
China mirrors.cloud.tencent.com Tecent Cloud Every day
Japan packagist.jp Hiraku forked Every 2 minutes Q4'14
Japan packagist.kawax.biz Kawax another Every hour Q4'18

🛑 Not working as a mirror of packagist.org (checked at Q1'20):

Location Mirror Maintainer Github Reason At least
China mirrors.huaweicloud.com Huawei Cloud Outdated Q3'19
China packagist.phpcomposer.com Outdated Q4'19

If you know any new mirror based or not on this one, please create a issue or a pull request with the new data.

Check status page for health mirror's.

World Map

This map shows working mirrors from above at the country level. The colors represent the topology drawn below.

🚀 Create your own mirror

Topology

💡Tip: use a machine with at least 2GB of RAM to avoid using the disk or swap space during sync.

⚠️ When syncing from DATA_MIRROR or MAIN_MIRROR, your server encodes and decodes all packages as .gz files to save disk space. You may need to enable server-side decoding for legacy composer clients that ask for decompressed packages.

There are currently three supported methods for creating your own mirror.

In all three methods, you need to clone the repository and copy .env.example to .env and modify to include your values instead of the defaults.

# Clone this repository
$ git clone https://github.com/websyther/packagist-mirror.git

# Setup environment variables
$ cd packagist-mirror
$ cp .env.example .env
$ nano .env

Docker Compose

Run the following commands to start a container for Nginx, PHP-FPM, and a worker that runs cron jobs.

# Start all Docker containers
$ docker-compose up -d

# Follow log output
$ docker-compose logs -f

Once the initial sync has finished, open https://localhost:9248 to see your site.

💡Tip: Add -f docker-compose.prod.yml between docker-compose and up or down while running the above commands. If you are using traefik, the services in this docker-compose file contain labels used by a running traefik container to automatically route traffic matching those labels to that container. It even auto-renews LetsEncrypt certificates for you.

Docker Nginx PHP

First, add the following line to /etc/crontab to tell the host to start a container for the packagist-mirror image on boot, replacing the values for each -e flag with your own. This will start the initial sync and generate the website files to be served by nginx.

Learn about more the available options for this docker image here.

* * * * * root docker run --name mirror --rm -v /var/www:/public \
-e MAINTAINER_REPO='packagist.com.br' \
-e APP_COUNTRY_NAME='Brazil' \
-e APP_COUNTRY_CODE='br' \
-e MAINTAINER_MIRROR='Webysther' \
-e MAINTAINER_PROFILE='https://github.com/Webysther' \
-e MAINTAINER_REPO='https://github.com/Webysther/packagist-mirror' \
-e URL='packagist.com.br' \
webysther/packagist-mirror

Next, add the following to /etc/nginx/sites-available/packagist.com.br.conf to host the website files:

server {
    index.html;

    server_name packagist.com.br www.packagist.com.br;

    location / {
        try_files $uri $uri/ =404;
        gzip_static on;
        gunzip on;
    }
}

To monitor sync progress, run the following command:

docker logs --follow --timestamps --tail 10 mirror

Nginx PHP

After cloning the repository, run the following commands to configure for your host.

$ cd packagist-mirror && composer install
$ cp .env.example .env

Then, schedule the command to create and update the mirror:

$ php bin/mirror create -vvv

Nginx will now serve your mirror at the configured URL.

🐧 Development & Contributing

Please see CONTRIBUTING and CONDUCT for details.

📋 Requirements

The following versions of PHP are supported by this version.

  • PHP >=7.2

🧪 Testing

$ vendor/bin/phpunit

🥂 Credits

💙 Other correlated projects

☮️ License

MIT License. Please see License File for more information.

FOSSA Status

packagist-mirror's People

Contributors

andreybolonin avatar baptistepillot avatar cwilby avatar dependabot-preview[bot] avatar dependabot-support avatar fossabot avatar hirak avatar jk2k avatar kkame avatar ktarasov avatar legoktm avatar mousavian avatar nikograno avatar olragon avatar pedrosoaresll avatar peter279k avatar scrutinizer-auto-fixer avatar seldaek avatar spotzero avatar stof avatar vrkansagara avatar webysther avatar wudi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

packagist-mirror's Issues

Build a miror behind a proxy

Hello,

Unless I missed something, but I can't build a mirror behind a proxy
In streamhandler.php there is a dns request with php_network_getaddress()

The proxy works well with env and curl export https_proxy=http://X.X.X.X:port

This feature could be very useful inside company without direct internet access

PHP warnings during initial mirror creation

$ php bin/mirror create
...
PHP Warning:  filemtime(): stat failed for ./public/index.html in /srv/packagist-mirror/src/Command/Create.php on line 406
PHP Warning:  unlink(./public/index.html): No such file or directory in /srv/packagist-mirror/src/Command/Create.php on line 407

How I can use an internal domain ?

When I tried to build a mirror with an internal DNS name, the composer request is broken because packages.json contains repo.packagist.org

I can use mydomain.int as mirror ?

LICENSE mismatch

The LICENSE file has the MIT license, but composer.json says CC0. Could you clarify which one it's supposed to be?

Also as a more minor note, the source files say "For the full license information, please view the LICENSE.md", except that file doesn't exist since it's just named LICENSE.

Change option create to ignore clean

php bin/mirror create --no-clean

To reduce IO for every update and keep for a while old packages metadata: usefull for high frequency mirror vs slow connection client.

This feature need #9 to be really useful.

"do not match its signature"

Hi,

I've mirrored a packagist repo sucessfully

$php webysthe-packagist-mirror/bin/mirror create -vvv
Loading providers from https://packagist.hesse.im/
packages.json updated

My private NGINX server shows me on https://private.repo.lan/packages.json :
{ "packages": [], "notify-batch": "https:\/\/packagist.org\/downloads\/", "providers-url": "\/p\/%package%$%hash%.json", "search": "https:\/\/packagist.org\/search.json?q=%query%&type=%type%", "list": "https:\/\/packagist.org\/packages\/list.json", "providers-api": "https:\/\/packagist.org\/providers\/%package%.json", "warning": "You are using an outdated version of Composer. Composer 2 is now available and you should upgrade. See https:\/\/getcomposer.org\/2", "warning-versions": "<1.99", "provider-includes": { "p\/provider-2013$%hash%.json": { "sha256": "d0dc77fd03e5732f1bdb80037c61df42ed94764bddb5f24774761c59739791cc" }, "p\/provider-2014$%hash%.json": { "sha256": "ab43f689a426163c7bd97eee1359d66270cf969532a976b03d4d4e608395b25c" }, "p\/provider-2015$%hash%.json": { "sha256": "5dd99e47c377393321c91a2b7cede7c1dff20f23728c89dd551c20543750cf5e" }, "p\/provider-2016$%hash%.json": { "sha256": "7b8f1749c41d2c755e25992d763588fa527308b20f53fb15523c5a938d960bc4" }, "p\/provider-2017$%hash%.json": { "sha256": "62c94621ae4a052b1d7e00cb9398703ef09bc8f69fe8fce22e309a831b6011f9" }, "p\/provider-2018$%hash%.json": { "sha256": "df4e171e392f024639a924670d6c2a243a645277d43fb3b141cef9efe7b9df12" }, "p\/provider-2019$%hash%.json": { "sha256": "166db562306e9fd8967eb173a9dbf601c65ce4a2e67bf3d18c132730f1f1f6f9" }, "p\/provider-2020$%hash%.json": { "sha256": "ff7e5318ac974ec8ce64a1d0232f038f9fbac3d987a2146719ff31c6fc578d40" }, "p\/provider-2020-07$%hash%.json": { "sha256": "22517059027c992f913d337d6b79f67b1f3305c1d969e99bfd8ec859e54d9866" }, "p\/provider-2020-10$%hash%.json": { "sha256": "9d48effa58bf64e0764d894494902ffdee59924ad8f63fe1126b315601a2cb9b" }, "p\/provider-2021-01$%hash%.json": { "sha256": "0d37334fff077049a79655aa7f4ecd4ad7a4c01b66a4fe4b24c8aa6170ae5789" }, "p\/provider-2021-04$%hash%.json": { "sha256": "070fc16ec215d0422e7791c4fb0e1279d00704fecaac221863b56f0d68bb3514" }, "p\/provider-archived$%hash%.json": { "sha256": "60d5f896e444581ddd4baae5eaf7f04b63ec618d00f1d8ddf3af1fcf41264015" }, "p\/provider-latest$%hash%.json": { "sha256": "be8f9b5a0f6e46b14e976d6f56ae297509644bbd11916bd90d6894aea4946d27" } } }

But when requiring diffferent libs, I got each time an error ; example with drush

$ composer require drush/drush

Using version ^10.5 for drush/drush
./composer.json has been created
Running composer update drush/drush
Loading composer repositories with package information
Installation failed, deleting ./composer.json.
[Composer\Repository\RepositorySecurityException]
The contents of https://private.repo.lan/p/consolidation/site-process%24f510087614b0f459ed763ade5338c9bbade5e69b1b574df465e4e13105e11643.json do not match its signature. This could indicate a man-in-the-middle attack or e.g. antivirus software corrupting files. Try running composer again and report this if you think it is a mistake.
require [--dev] [--dry-run] [--prefer-source] [--prefer-dist] [--prefer-install PREFER-INSTALL] [--fixed] [--no-suggest] [--no-progress] [--no-update] [--no-install] [--no-scripts] [--update-no-dev] [-w|--update-with-dependencies] [-W|--update-with-all-dependencies] [--with-dependencies] [--with-all-dependencies] [--ignore-platform-req IGNORE-PLATFORM-REQ] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [--sort-packages] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--apcu-autoloader-prefix APCU-AUTOLOADER-PREFIX] [--] []...

What's wrong with my mirror ??
Thanks a lot :)

package.json is being downloaded as binary file

Hi there, I'm trying to use the brazilian mirror but I'm aways getting this when trying to run composer require:

...
"https://packagist.com.br/packages.json" does not contain valid JSON
Parse error on line 1:                                                                                                                                                                          
\\\\\\Qo                                                                                                                                                                                        
                                                                                                                                                                                                
^                                                                                                                                                                                               
Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['                                                                                                                          
https://packagist.com.br could not be fully loaded, package information was loaded from the local cache and may be out of date

I tried to fetch the URL directly from curl and the return was this:

danielcosta:~ danielcosta$ curl -I https://packagist.com.br/packages.json
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Thu, 06 Sep 2018 17:28:58 GMT
Content-Type: application/json
Content-Length: 683
Last-Modified: Thu, 06 Sep 2018 17:28:40 GMT
Connection: keep-alive
ETag: "5b9163c8-2ab"
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: DENY
Accept-Ranges: bytes

danielcosta:~ danielcosta$ curl https://packagist.com.br/packages.json
Qo
  )
cy_e)Q |nw/C0L>p''ힵ_}}onkOq+}y8xұ=s>OߏNoBrwp겹'k?~      ێ3^0
                                                            pP*|EH
##=עsfI5p4'
BD8f*yam??'M^i"&\P2A6f%,qXM=TK4e;v+JN!u6(6F]p[8X]U@ڵ1gBhJ\0.3{C(tHK4J3Vp.VsSDγ^*fEr.fˎ,)^cNFGX>L1$eX
                                                                                                    =6ZOsr1Jk
danielcosta:~ danielcosta$ 

Do you have any idea why is this happening?

I tried the commands from two different machines and connections, is this a general issue or is it only happening with me?

Extra dependencies?

Maybe I missed something, but I don't see where illuminate/support and nesbot/carbon are being used.

Add i8n

Better alternative to fork-change files style (sample #117)

Storage and access

@webysther Thanks for this project.

  1. I'm still trying to work out the differences between this and satis. Is packagist-mirror "just" enabling a geographical close copy while satis has a larger scope?
  2. In terms of architecture, would the result of packagist-mirror just be pushed to S3 and served via something like Cloudflare?

Services lose get parameters after mirroring

Get parameters in services are lost during the mirroring:

# https://packagist.org/packages.json
search: "https://packagist.org/search.json?q=%query%&type=%type%",

# mirrored version
search: "https://packagist.org/search.json"

I think the issue is in src/Provider.php
(not tested myself yet.)
https://github.com/Webysther/packagist-mirror/blob/9c6d9b4a735d59fd34616f346359228d222b84ad/src/Provider.php#L109-L119
It seems to me that during path_url only the path itself is kept, get parameters are lost.

Error - Impossible to create the root directory

An error "Impossible to create the root directory" occurred during the installation.

$ cd public/p
$ ls -1 | wc -l
32825
$

This is a limit subfolder on OS. Can do subfolder of the form public/p/0/..., public/p/1/.., ... , public/p/z/..

I don’t want to switch to ZFS. Knowledge of PHP is still not enough to do it yourself.

OS: FreeBSD 12 (i386 and amd64)
FileSystem: UFS

Not Downloading Meta information properly

Today morning i setup a digitalocean server to mirror packages in my nearby country / place

but it stopped downloading files after 2015
image

and to download meta for (2013,2014,2015) it almost took around 4+ hrs and sometimes the process just kills it self. i need to do it manually :-(

Will it mirror zip files of packages?

I found that packagist place the zip files of packages in github, if this mirror only sync json files without package files, that if is limit used, because download file from github is also slow.

Update mirroring code to our official mirror code

I just published https://github.com/composer/mirror which supports Composer 1 and Composer 2 style metadata mirroring. I would appreciate very much if you could update this repo to use this instead of however you are doing it now. I am fairly sure our implementation is correct, and especially that it will be maintained.

I'm happy to discuss how to achieve this etc.

Also note that generally speaking I hope Composer 2 will reduce the need for third party mirrors as it does parallelize requests to the repository, and data transfer is overall smaller, so latency should be much less problematic.

JSON encrypted/compressed over HTTPS

After deploying this app to a DigitalOcean droplet I was able to connect my local composer instance and install dependencies.

After enabling HTTP on the server using LetsEncrypt, composer is no longer able to fetch package.json as it appears encrypted/compressed.

I have tried enabling/disabling gzip to no avail.

Update: the file is encrypted/encoded on the server when opened with sudo nano package.json.

packagist.hostuj.to down

I'm getting packagist.hostuj.to refused to connect.. Recommending to remove this mirror from the list.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.