GithubHelp home page GithubHelp logo

luxysiv / cloudflare-gateway-pihole Goto Github PK

View Code? Open in Web Editor NEW
66.0 1.0 110.0 1.94 MB

Make ad blocking dns using Cloudflare Gateway Zero Trust

Python 100.00%
ad-blocking cloudflare-gateway dns dns-blocking

cloudflare-gateway-pihole's Introduction

CF_logo_stacked_whitetype

English | Việt Nam

Pihole styled, but using Cloudflare Gateway

For Devs, Ops, and everyone who hates Ads.

Create your ad blocklist using Cloudflare Gateway.

Credit goes there


Thanks a lot to @nhubaotruong for his contributions.

Readme by @minlaxz.

Added dynamic domain filter (whitelist and blacklist) idea (please check ini files, as you may also need to modify those).

Added dynamic domain filter (whitelist and blacklist) to Actions variables (please check dynamic_blacklist.txt and dynamic_whitelist.txt. to know examples to add Value*).Use DYNAMIC_BLACKLIST and DYNAMIC_WHITELIST for Name* in Actions variables

Supported styles


https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt

or

[Ad-Urls]
Adguard = https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt

Custom URLs


  • Add to file:

    White list whitelist.ini and block list adlist.ini.

  • Add to GitHub Action variables:

    Name*

    ADLIST_URLS or WHITELIST_URLS.

    Value* URLs list

    Example:

    https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt
    https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/light-onlydomains.txt
    
  • You should add your ad list and whitelist to Action variables. If you update your fork, your custom list will not be lost.

How to set this up?


  1. Fork this repository to your account.

  2. Grab your Cloudflare Account ID (found after https://dash.cloudflare.com/) from ➞ https://dash.cloudflare.com/?to=/:account/workers.

  3. Create your API Token from ➞ https://dash.cloudflare.com/profile/api-tokens with 3 permissions:

    1. Account.Zero Trust : Edit
    2. Account.Account Firewall Access Rules : Edit
    3. Account.Access: Apps and Policies : Edit
  4. Add Repository Secrets to your forked repository: ➞ https://github.com/<username>/<forked-repository>/settings/secrets/actions

    1. Set Cloudflare Account ID to CF_IDENTIFIER.
    2. Set API Token to CF_API_TOKEN.

Schedule


Due to a limited 2-month commitment from GitHub Actions, you can create and paste this code to run on Cloudflare Workers. Remember, GitHub Tokens generate with no expiration and all permissions.

addEventListener('scheduled', event => {
  event.waitUntil(handleScheduledEvent());
});

async function handleScheduledEvent() {
  const GITHUB_TOKEN = 'YOUR_GITHUB_TOKEN_HERE';
  try {
    const dispatchResponse = await fetch('https://api.github.com/repos/YOUR_USER_NAME/YOUR_REPO_NAME/actions/workflows/main.yml/dispatches', {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${GITHUB_TOKEN}`,
        'Content-Type': 'application/json',
        'User-Agent': 'Mozilla/5.0',
      },
      body: JSON.stringify({
        ref: 'main'
      }),
    });

    if (!dispatchResponse.ok) throw new Error('Failed to dispatch workflow');
  } catch (error) {
    console.error('Error handling scheduled event:', error);
  }
}

Remember to set up Cloudflare Workers triggers.

How to set up using Termux?


To use this tool on the GOAT Termux, follow the steps below. If you are already familiar with setting up Python and the basics, you can skip this section.

Method 1:

  1. Open Termux and run the following commands one by one:
yes | pkg upgrade
yes | pkg install python-pip
yes | pkg install git
# Clone your forked repo #
git clone https://github.com/<username>/<repo-name>.git
  1. Navigate to the cloned repository folder:
cd <repo-name>
  1. Edit the .env file (required):
nano .env

After editing, press CTRL + X, then Y, and ENTER to save the file.

  1. Run the command to upload (update) your DNS list:
python -m src run
  1. Run the command to delete your DNS list:
python -m src leave 

Method 2:

  1. Download the ZIP file of the repository from the 'Code' button on the GitHub page and select 'Download ZIP'.

  2. Unzip the downloaded file.

  3. Edit the values in .env and adlist.ini etc...

  4. Open Termux and enter the following commands to set up Python and necessary tools:

yes | pkg upgrade
yes | pkg install python-pip
termux-setup-storage
  1. Allow Termux to access storage.

  2. Navigate to the folder containing the unzipped source code:

cd storage/downloads/Cloudflare-Gateway-Pihole-main
  1. Run the command to upload (update) your DNS list:
python -m src run
  1. Run the command to delete your DNS list:
python -m src leave

If you encounter issues during setup, you can refer to termux-change-repo for changing Termux repositories.

Note


  • The limit of Cloudflare Gateway Zero Trust free is 300k domains, so remember to pay attention to the workflow logs. If it is exceeded, the script will stop.

  • If you have uploaded lists using another script, you should delete them using the delete feature of the uploaded script or delete them manually.

  • I have updated the feature to delete lists when you no longer need to use the script. Go to main.yml as follows:

      - name: Cloudflare Gateway Zero Trust 
        run: python -m src leave

Note from @minlaxz:

  1. Domain list style: I personally preferred the second one in blacklist styles, which is more readable and concise.
  2. Dynamic domain list: You can also update your dynamic (fluid) whitelist and blacklist using dynamic_blacklist.txt and dynamic_whitelist.txt.
  3. Deprecated using .env: Setting sensitive information inside a public repository is considered too dangerous, since any unwanted person could easily steal your Cloudflare credentials from that .env file.

🥂🥂 Cheers! 🍻🍻

cloudflare-gateway-pihole's People

Contributors

duc2ht avatar luxysiv avatar minlaxz avatar nhubaotruong avatar rhoggs-bot-test-account avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

cloudflare-gateway-pihole's Issues

Suggestions

Hey! I did all like in instructions but nothing happened in my account. Maybe I miss something? I changed .env file to my ID data too. But new rules don't appear in my filter list folder.

URL scraping error

2024-06-20 05:47:53.141 | INFO | src:info:89 - Number of whitelisted domains: 5
2024-06-20 05:47:58.193 | INFO | src:info:89 - Number of blocked domains: 253825
2024-06-20 05:47:58.289 | INFO | src:info:89 - Number of final domains: 253822
2024-06-20 05:47:58.650 | INFO | src:info:89 - Total lists on Cloudflare: 118
2024-06-20 05:47:58.650 | INFO | src:info:89 - Total domains on Cloudflare: 117274
2024-06-20 05:47:58.822 | INFO | src:info:89 - Total chunked lists generated: 254
2024-06-20 05:47:59.018 | INFO | src:info:89 - Updating list [AdBlock-DNS-Filters] - 001
2024-06-20 05:47:59.757 | INFO | src:info:89 - Updating list [AdBlock-DNS-Filters] - 002
2024-06-20 05:48:00.767 | INFO | src:info:89 - Updating list [AdBlock-DNS-Filters] - 003
2024-06-20 05:48:01.835 | INFO | src:info:89 - Updating list [AdBlock-DNS-Filters] - 004
2024-06-20 05:48:02.890 | INFO | src:info:89 - Updating list [AdBlock-DNS-Filters] - 005
2024-06-20 05:48:03.408 | INFO | src:info:89 - Retrying (1): 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/accounts/***/gateway/lists/90cfd341-8f9a-40b3-aa8b-efba858bc178
2024-06-20 05:48:03.408 | INFO | src:info:89 - Sleeping before next retry (1)
2024-06-20 05:48:04.884 | INFO | src:info:89 - Retrying (2): 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/accounts/***/gateway/lists/90cfd341-8f9a-40b3-aa8b-efba[85](https://github.com/asterriya/Cloudflare-Gateway-Pihole/actions/runs/9592280677/job/26450546403#step:5:86)8bc178
2024-06-20 05:48:04.884 | INFO | src:info:89 - Sleeping before next retry (2)
2024-06-20 05:48:06.595 | INFO | src:info:89 - Retrying (3): 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/accounts/***/gateway/lists/90cfd341-8f9a-40b3-aa8b-efba858bc178
2024-06-20 05:48:06.596 | INFO | src:info:89 - Sleeping before next retry (3)
2024-06-20 05:48:08.272 | INFO | src:info:89 - Retrying (4): 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/accounts/***/gateway/lists/[90](https://github.com/asterriya/Cloudflare-Gateway-Pihole/actions/runs/9592280677/job/26450546403#step:5:91)cfd341-8f9a-40b3-aa8b-efba858bc178
2024-06-20 05:48:08.272 | INFO | src:info:89 - Sleeping before next retry (4)
2024-06-20 05:48:11.663 | INFO | src:info:89 - Retrying (5): 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/accounts/***/gateway/lists/90cfd341-8f9a-40b3-aa8b-efba858bc178
2024-06-20 05:48:11.663 | INFO | src:info:89 - Sleeping before next retry (5)
2024-06-20 05:48:22.166 | INFO | src:info:89 - Retrying (6): 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/accounts/***/gateway/lists/90cfd341-8f9a-40b3-aa8b-efba858bc178
2024-06-20 05:48:22.166 | INFO | src:info:89 - Sleeping before next retry (6)
2024-06-20 05:48:32.694 | INFO | src:info:89 - Retrying (7): 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/accounts/***/gateway/lists/90cfd341-8f9a-40b3-aa8b-efba858bc178
2024-06-20 05:48:32.694 | INFO | src:info:89 - Sleeping before next retry (7)
2024-06-20 05:48:43.140 | INFO | src:info:89 - Retrying (8): 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/accounts/***/gateway/lists/90cfd341-8f9a-40b3-aa8b-efba858bc178
2024-06-20 05:48:43.141 | INFO | src:info:89 - Sleeping before next retry (8)
2024-06-20 05:48:44.977 | INFO | src:info:89 - Retrying (9): 400 Client Error: Bad Request for url: https://api.cloudflare.com/client/v4/accounts/***/gateway/lists/90cfd341-8f9a-40b3-aa8b-efba858bc178
2024-06-20 05:48:44.978 | INFO | src:info:89 - Sleeping before next retry (9)

update adblock list

How to setup auto update for the workflow so that adblock lists keeps updated with latest.

List uploading stop working

Here is a log:

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 148, in
asyncio.run(main())
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 145, in main
await cloudflaremanager.run()
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 98, in run
cf_lists = await asyncio.gather(*create_list_tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/cloudflare.py", line 14, in wrapper
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/cloudflare.py", line 43, in create_list
raise Exception("Failed to create Cloudflare list")
Exception: Failed to create Cloudflare list
Error: Process completed with exit code 1.

And this error happens almost for all lists in domains / host format. But yesterday all works fine.

More language

@s0uboxxx bạn có thể tạo PR Readme tiếng Nhật không? Bạn làm 1 file giống vi.md trong docs và thêm link truy cập giống README.md.

Block ads only for a specific location

Hello,

Can the functionality of limiting this to specific locations be added?

This is already possible in firewall policies to some extent but the firewall logic needs to be modified:

Screenshot

fork update issue

Hi,

Please fix this issue, so we can update normally.

This branch has conflicts that must be resolved
Discard 112 commits to make this branch match the upstream repository. 112 commits will be removed from this branch.

You can resolve merge conflicts using the command line and a text editor.

Python issue

Run python -m src run
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 2, in
from src import (
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/utils.py", line 3, in
from src import (
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/cloudflare.py", line 2, in
from tenacity import (
File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/tenacity/init.py", line 653, in
from tenacity.asyncio import AsyncRetrying # noqa:E402,I100
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'tenacity.asyncio'
Error: Process completed with exit code 1.

I am getting this since 5 days. Before everything was running fine.

          I am getting this since 5 days. Before everything was running fine.

Run python -m src
2024-06-10 08:50:41.740 | INFO | src:info:50 - Downloaded file from https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/light-onlydomains.txt File size: 1244590
2024-06-10 08:50:42.010 | INFO | src:info:50 - Downloaded file from https://gitlab.com/fusionjack/adhell3-hosts/-/raw/master/hosts File size: 218535
2024-06-10 08:50:43.655 | INFO | src:info:50 - Downloaded file from https://abpvn.com/android/abpvn.txt File size: 600492
2024-06-10 08:50:43.668 | INFO | src:info:50 - Downloaded file from https://raw.githubusercontent.com/jerryn70/GoodbyeAds/master/Extension/GoodbyeAds-Xiaomi-Extension.txt File size: 9218
2024-06-10 08:50:43.751 | INFO | src:info:50 - Downloaded file from https://raw.githubusercontent.com/nextdns/native-tracking-domains/main/domains/xiaomi File size: 217
2024-06-10 08:50:43.785 | INFO | src:info:50 - Downloaded file from https://raw.githubusercontent.com/jerryn70/GoodbyeAds/master/Extension/GoodbyeAds-Samsung-AdBlock.txt File size: 4437
2024-06-10 08:50:43.869 | INFO | src:info:50 - Downloaded file from https://raw.githubusercontent.com/nextdns/native-tracking-domains/main/domains/samsung File size: 92
2024-06-10 08:50:44.237 | INFO | src:info:50 - Downloaded file from https://malware-filter.gitlab.io/malware-filter/vn-badsite-filter-hosts.txt File size: 374452
2024-06-10 08:50:44.286 | INFO | src:info:50 - Downloaded file from https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt File size: 1417556
2024-06-10 08:50:45.092 | INFO | src:info:50 - Downloaded file from https://small.oisd.nl/domainswild2 File size: 840485
2024-06-10 08:50:45.093 | INFO | src:info:50 - Number of whitelisted domains: 1
2024-06-10 08:50:45.881 | INFO | src:info:50 - Number of blocked domains: 122503
2024-06-10 08:50:45.916 | INFO | src:info:50 - Number of final domains: 122502
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 86, in
cloudflare_manager.run()
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 41, in run
current_lists["result"].sort(key=lambda x: int(re.search(r'\d+', x["name"]).group()))
File "/home/runner/work/Cloudflare-Gateway-Pihole/Cloudflare-Gateway-Pihole/src/main.py", line 41, in
current_lists["result"].sort(key=lambda x: int(re.search(r'\d+', x["name"]).group()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'
Error: Process completed with exit code 1.

Originally posted by @axelweichert in #37 (comment)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.