pi-hole / pi-hole Goto Github PK

If there is blank line in blacklist.txt, these lines are counted as blacklisted domains which is wrong.
In my case, I have 35 lines among then 4 empty lines but reported as

** Neutrino emissions detected...
Skipping adaway.org list because it does not have any new entries...
Skipping adblock.gjtech.net list because it does not have any new entries...
Skipping adblock.mahakala.is list because it does not have any new entries...
Skipping hosts-file.net list because it does not have any new entries...
Skipping www.malwaredomainlist.com list because it does not have any new entries...
Skipping pgl.yoyo.org list because it does not have any new entries...
Skipping someonewhocares.org list because it does not have any new entries...
Skipping winhelp2002.mvps.org list because it does not have any new entries...
** Aggregating list of domains...
** Blacklisting 35 domain(s)...
** 398451 domains being pulled in by gravity...
** 377619 unique domains trapped in the event horizon.
** Formatting domains into a HOSTS file...

Make a script that will verify functionality of the Pi-hole so users can see if it is working properly.

There is no easy way for existing Pi-hole users to update their gravity.sh script and Pi-hole support files or the Web interface. I would like to get some sort of mechanism installed that can query the repos and install updates.

I have found that some sites I visit just don't work because they are blocked, so I want a quick way to whitelist a domain. It would be a separate command that could quickly remove one or more domains from /etc/hosts.

So it might look something like this:

~$ whitelist.sh somedomaintowhitelist.com

and the output like:

** 1 domain whitelisted
374874 domains are still blocked.

I uncommented the line that determined the IP dynamically, and it worked mostly.

But there was one case where it happened to run (via cron) and come up with a blank IP... not sure why or how... but it corrupted the dns config file and broke my entire network.

It may have been due to a temporary problem with my connection or cable.

After that nothing works.

Proposal: Add a check for a blank IP address in the case of automated detection. I've changed to a static IP in the meantime since this is better for my case anyway...

The Pi-hole redirects all requests to the blank HTML file. So if you try to go to the dashboard page, you can't get to it.

Some ideas to fix this:

• Put some sort of if/else logic into the lighttpd.conf file (not sure how to do this just yet)
• Use a different port

According to commit e1d79ff a 1x1 png i served instead of a blank html page but it's not on my install. Do I have to add it myself and if so, how? Did I miss something?

Issue in current version of loops

current code is as below:
Remove lines with no dots (i.e. localhost, localdomain, etc)
echo -n "" > $origin/$supernova;for i in $origin/*.$justDomainsExtension;do grep '.' $i >> $origin/$supernova;done

Here the source file should be andLight.txt file not the list of .domains files as they are already included in andLight.txt
In addition, blacklist.txt is in matter.txt and in andLight.txt but not in any .domain file

Create a .deb of the pi-hole package to let facilitate an easier end-user experience. Create a repository for distribution of the .deb via apt-get on Debian-like environments.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

sudo service dnsmasq restart removed in the master trunk
it used to be OK in the loop trunk

if no service restart, the weekly cron of gravity.sh will not update the new lists.

My /etc/pihole directory contains 3 files:

LICENSE
list.0.adaway.org.domains
list.1.adblock.gjtech.net.domains

I did the automated setup using DietPi. Any help? Can I just cat these two lists into gravity.list, or is there some finessing involved before the addresses are added?

Herebelow the output of ifconfig
note that Raspberry is set with french configuration
output contains "inet adr" instead of "inet addr" which causes an error in gravity_adv.sh
piholeIP=$(ifconfig eth0 | awk '/inet addr/{print substr($2,6)}')

pi@raspberrypi ~ $ ifconfig eth0
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet adr:192.168.1.xx Bcast:192.168.1.255 Masque:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:192676 errors:0 dropped:1 overruns:0 frame:0
TX packets:161849 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:1000
RX bytes:135021006 (128.7 MiB) TX bytes:22700864 (21.6 MiB)

First, terrific utility and a nice way to use an extra RPi I had around.

I have the system up and running and it works pretty well. But everyonce in a while I'll open a page and the ads are all there, quite shocking to see when it happens after the relative bliss of clear pages.

However, usually if I refresh the page, they will disappear. Then if I refresh many times, they'll sometimes reappear and disappear again.

I'm sure something is choking the RPi somewhere and causing the secondary DNS to be used, but I'm not sure how to troubleshoot this.

For the static ip in jessie (not 100% sure why this distro is recommending configuring static ip's this way, maybe because it's desktop oriented?)

For me to get it to work, I had to ensure the primary interface was set to manual:

#/etc/network/interfaces
...
auto eth0
allow-hotplug eth0
iface eth0 inet manual
...

I then added the following to the bottom of the default /etc/dhcpcd.conf file and rebooted
(change 192.168.1.254/24 to your ip/subnet of choice / and use your preferred fallback dns)

#/etc/dhcpcd.conf
...

interface eth0
static ip_address=192.168.1.254/24
static routers=192.168.1.1
static domain_name_servers=8.8.8.8 8.8.4.4

The gravity.sh script in the article wrote to /etc/pihole/gravity.list and not to /etc/hosts, so I added addn-hosts=/etc/pihole/gravity.list to fix:

# /etc/dnsmasq.conf
domain-needed
bogus-priv
no-resolv
server=8.8.8.8
server=8.8.4.4
interface=eth0
listen-address=127.0.0.1
cache-size=10000
log-queries
log-facility=/var/log/pihole.log
local-ttl=300
addn-hosts=/etc/pihole/gravity.list

and after a

 sudo systemctl restart dnsmasq

all is well now... Thanks!

Not really an issue.

I just ported the web interface to Go. https://github.com/girishso/pi-hole-web single executable with all the assets and web server included. No lighttpd or php-fpm required! :)

Hi,

I noticed that with pi-hole as my primary dns enable, I can't interact with javascript menus and links in various websites. If I temporarily change my dns settings and visit the website, everything goes back to normal. Here is a universally accessible website as an example:
http://www.scholl.co.uk/home/foot-care/insoles-and-arch-supports/
If I visit the website with pi-hole enabled, the button at the top acknowledging the use of cookies cannot be clicked, and all the menus in the page link to #
I tried to whitelist the websites, but it does not help. Any suggestions?

First of all kudo's for an excellent piece of software!

My issue is that I can't login to Steam when pi-hole is handling the DNS.

When I disable pi-hole I get through immediatly.

I don't really know where to start looking. I'm willing to investigate myself if you have some pointers.

Many users are having trouble downloading the mahakala list and running out of memory. Giving the GPU 16MB seems to work for a lot of people, but as the list grows larger, I'm looking for a long-term solution.

I'm thinking it would be a good idea for the Pi-Hole to include a DHCP server so that it can give machines the correct DNS settings when they connect to the network.

I'll happily hack about with this if you think it's a good idea.

Cheers,

R.

As screenshot shows site is added to blacklist, ping is adblock but ads are still getting through...

Browser i MS Edge.

I've cleared EVERYTHING in edge... any ideas or things that would be worth testing?

I was getting the following errors when updating the database:

pi@pihole ~ $ sudo /usr/local/bin/gravity.sh
** Neutrino emissions detected...
Skipping adaway.org list because it does not have any new entries...
Skipping adblock.gjtech.net list because it does not have any new entries...
Getting adblock.mahakala.is list...
/usr/local/bin/gravity.sh: fork: Cannot allocate memory

I added 512 MB's of Swap and that seemed to fix this issue.

Using grep -v -x -f on the whitelist doesn't seem to always remove the domain names from the ad list. I'm not sure why yet...

Hello,

I was taking a look through the dnsmasq documentation and I would like to log the rquester IP using the --log-queries=extra option as described:

-q, --log-queries
Log the results of DNS queries handled by dnsmasq. Enable a full cache dump on receipt of SIGUSR1. If the argument "extra" is supplied, ie --log-queries=extra then the log has extra information at the start of each line. This consists of a serial number which ties together the log lines associated with an individual query, and the IP address of the requestor.

But when I change my /etc/dnsmasq.conf to this:

addn-hosts=/etc/pihole/gravity.list
domain-needed
bogus-priv
no-resolv
server=8.8.8.8
server=8.8.4.4
interface=eth0
listen-address=127.0.0.1
cache-size=10000
log-queries=extra
log-facility=/var/log/pihole.log
local-ttl=300

And I try to restart using sudo service dnsmasq restart, I get this:

[FAIL] Restarting DNS forwarder and DHCP server: configuration syntax check failed!

Any thoughts that could help me here?

Thank you!

hi

I installed your adblocking script on a unix based machine.
I previously installed using your old guide and it worked perfectly.
Now I am trying to use the newer guide and am falling short.
I've followed the guide but when testing (ping 247media.com) and dig a-ads.com I am failing.
the ping command doesn't return anything and the dig command just returns the actual site and is not redirected to the "pihole".

ping : PING 247media.com (54.221.208.242) 56(84) bytes of data.
and it hangs until I cancel...

dnsmasq is running but returning
/etc/resolvconf/update.d/libc: Warning: /etc/resolv.conf is not a symbolic link to /etc/resolvconf/run/resolv.conf
when I restart the service

dnsmasq.conf is:

domain-needed
bogus-priv
no-resolv
server=8.8.8.8
server=8.8.4.4
interface=eth0
listen-address=127.0.0.1
cache-size=10000
log-queries
log-facility=/var/log/pihole.log
local-ttl=300

I tried creating the resolv.conf file but that didn't change anything..

Any help is much appreciated!

Getting adaway.org list... Skipping list because it does not have any new entries.
Getting adblock.gjtech.net list... Skipping list because it does not have any new entries.
Getting adblock.mahakala.is list... Killed

What could be wrong?

Hi, I successfully installed pi-hole on Raspberry (B) for the first time, just to test it out, then I formated SD card to fresh image a now when I run installation script I always get

[....] Restarting DNS forwarder and DHCP server: dnsmasq
dnsmasq: cannot fork into background: Cannot allocate memory
 failed!
Restarting services...
No resolv.conf for interface lo.dnsmasq
[....] Restarting DNS forwarder and DHCP server: dnsmasq
dnsmasq: cannot fork into background: Cannot allocate memory
 failed!
[ ok ] Stopping web server: lighttpd.
[ ok ] Starting web server: lighttpd.

I also tried on B+ with same result. Any ideas?

My router (Fritz!Box) allows me to set the DNS servers to be used for name resolution so the router acts as a DNS proxy. I use the (local) Raspberry Pi as the DNS server for the router and this works, except on the devices connected to the router the domains it should block do not resolve at all instead of resolving to the Raspberry Pi. When I configure my devices to use the Raspberry Pi as a DNS server directly the domains do resolve. Does anyone else have the same issue?

Sometimes entries from the third-party lists have errors, such as adlog..com (two periods instead of one).

This will probably require some sed and awk skills to check for the proper formatting of:

subdomain(s) (if applicable), a period (.), the domain, another period (.), and finally, the top level domain.

I have installed Pi-Hole and can access the interface, but I can't interact with the more info buttons on the GUI.

I manually added the pull request from iamdanielv to gravity.sh and now it works fine when running as sudo, it finds the whitelist.

Won't work at all if I run it without sudo, then I get file access errors and failure to restart dnsmasq etc.

This is with the latest Raspian with the latest updates and logged on as default user "pi"...

Long term goal of creating a Pi-Hole Distro for user one-shot install from blank SD-Card to full ad-blocker (in less time than it takes to have a pizza delivered.)

Hi,
I'm geting a runtime error when running chronometer.sh

Runtime error (func=(main), adr=3): Divide by zero
(standard_in) 1: syntax error

any idea?

I found that the whole of xkcd.com was blocked of all their image content, and I NEED my xkcd.

I've hacked a quick fix for myself, and this is a quick start for other people:

# Sort the aggregated results and remove any duplicates
# AND Apply whitelist
echo "Removing duplicates and formatting to address=/<ad domain>/"$piholeIP
cat /tmp/matter.txt | sed $'s/\r$//' | sort | uniq | sed '/^$/d' | cat - /usr/local/etc/whitelist.txt /usr/local/etc/whitelist.txt | sort | uniq -u |
 awk -v "IP=$piholeIP" '{sub(/\r$/,""); print "address=/"$0"/"IP}' > /tmp/andLight.txt

EDIT: Added whitelist twice to prevent accidental addition again in some cases.

Hey, not a big problem but you can skip `sort' calls in every curl - you're sorting anyway later when you're looking for duplicates.

Also, not sure what you're trying to do here:
grep -v "#" | sed '/^$/d' | sed 's/\ /\ /g' | grep -v '^' | grep -v '$' | awk '{print $2}' | grep -v '^' | grep -v '$'

but this is quite overkill - how about:
awk '{if ($1 !~ "#" && $1 !~ "/" && $2 !~ "#" && $2 !~ "/" && $0 != "^$" && $2 != "") { print $2}}'

If i'm seeing right, those files are similar to /etc/hosts (ip hostname, and comments (# and / based ?)) and you're only interested in 2nd column if line is valid - if so this should do it (might need few more conditions)

In fact you could 'dump' all those files to one file and process it in end - will be much faster.

Then, /tmp/matter.txt and /tmp/andLight.txt could be moved to /dev/shm (ramfs) so you do not operate on SD card (unless /tmp is already ram based - did not checked for a while how it is in Pi now)

Hey,

So I'm not exactly sure what I did wrong

• Clean install of raspbian jessie
• Ran curl -L install.pi-hole.net | bash
• Configured ip address to be one outside my dhcp pool
• Restarted pi
• Changed my DNS server on the router to the pi.

As it turns out both my phone (Android) and my computer (OS X) hated it and gave the following message:

DNS_PROBE_FINISHED_BAD_CONFIG

The server at mashable.com can't be found, because the DNS lookup failed. DNS is the network service that translates a website's name to its Internet address. This error is most often caused by having no connection to the Internet or a misconfigured network. It can also be caused by an unresponsive DNS server or a firewall preventing Google Chrome from accessing the network.
Check your DNS settings
Contact your network administrator if you're not sure what this means.
Try disabling network prediction
Go to the Chrome menu > Settings > Show advanced settings... and deselect "Prefetch resources to load pages more quickly." If this does not resolve the issue, we recommend selecting this option again for improved performance.
Allow Chrome to access the network in your firewall or antivirus settings.
If it is already listed as a program allowed to access the network, try removing it from the list and adding it again.

Suggestions on what I may of missed?
Router in use is a ZTE ZXHN H298N

It would be nice and easy to have a whitelist editor on the web interface!

Your script https://github.com/jacobsalmela/pi-hole/blob/master/automated%20install/basic-install.sh ends with rebooting the whole box - is this needed indeed? Why not just restart the services, since they have already been stopped before modifying the config files...?

It seems a bunch of lines in this file are corrupted, I think there should be a basic check to make sure that only normally characters are allowed in.

If you try to whitelist a domain using the standalone script, but do not have a whitelist.txt file, a grep error is produced because the file is not there.

Whitelisting someaddomain.com...
grep: /etc/pihole/whitelist.txt: No such file or directory
Modifying hosts file...
** 1 domain(s) whitelisted.

The version of lighttpd that can be installed via apt-get in the latest version of Raspbian ("Jessie"), is 1.4.35 and the root folder has changed. Instead of the files being in /var/www they are in /var/www/html. I will need to update the installer script to detect what version a user is running and then install the files in the correct place.

Symptom:

Lighttpd reports "WARNING: unknown config-key: url.access-deny (ignored)" when starting or restarting.

Reason:

Lighttpd is not being told to use "mod_access" during startup.

Solution:

server.modules section in /etc/lighttpd/lighttpd.conf should read:

server.modules = (
        "mod_access",
        "mod_expire",
        "mod_compress",
        "mod_redirect",
        "mod_setenv",
        "mod_rewrite"
)

I get this error:
pi@rp1 /usr/local/bin $ sudo sh gravity.sh
gravity.sh: 5: gravity.sh: Syntax error: newline unexpected

This is raspberry pi2 with raspbian
pi@ rp1 /usr/local/bin $ cat /proc/version
Linux version 3.18.11+ (dc4@dc4-XPS13-9333) (gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1+bzr2650 - Linaro GCC 2014.03) ) #781 PREEMPT Tue Apr 21 18:02:18 BST 2015

pi@rp1 /usr/local/bin $ cat /proc/cpuinfo
processor : 0
model name : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS : 2.00
Features : half thumb fastmult vfp edsp java tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xb76
CPU revision : 7

Hardware : BCM2708
Revision : 0010
Serial : 0000000040f0a897

pi@rp1 /usr/local/bin $ sudo rm gravity.sh
pi@rp1 /usr/local/bin $ ls
chronometer.sh indiecity
pi@rp1 /usr/local/bin $ sudo wget https://github.com/jacobsalmela/pi-hole/blob/master/gravity.sh
--2015-08-27 13:25:31-- https://github.com/jacobsalmela/pi-hole/blob/master/gravity.sh
Resolving github.com (github.com)... 192.30.252.129
Connecting to github.com (github.com)|192.30.252.129|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `gravity.sh'

[ <=>                                                                                                                  ] 74,921      --.-K/s   in 0.1s    

2015-08-27 13:25:37 (562 KB/s) - `gravity.sh' saved [74921]

pi@rp1 /usr/local/bin $ sudo sh gravity.sh
gravity.sh: 5: gravity.sh: Syntax error: newline unexpected
pi@rp1 /usr/local/bin $ ls -ls
total 84
4 -rwxr-xr-x 1 root staff 1392 Aug 12 22:35 chronometer.sh
76 -rw-r--r-- 1 root staff 74921 Aug 27 13:25 gravity.sh
4 drwxrwsr-x 2 root indiecity 4096 May 6 18:17 indiecity
pi@rp1 /usr/local/bin $
pi@ rp1 /usr/local/bin $ uname -a
Linux eautapi 3.18.11+ #781 PREEMPT Tue Apr 21 18:02:18 BST 2015 armv6l GNU/Linux

This only happens in a few circumstances, but as an example, say you used the mahakala list at one point and then decided you did not want it anymore so you either comment it out or remove it from the sources array and then run gravity.sh to apply the changes.

This doesn't work because the list.n.adblock.mahakala.is.domains file already exists in /etc/pihole/ from when the script ran previously. So when the find command searches for any .domains file to use to create the aggregated list, it will still apply the mahakala list because that file is still there.

So some sort of logic needs to exist that verifies that the domains in the sources array match the files that exist in /etc/pihole/ so only the correct ones apply.

Another instance when this happened is when a user added multiple new sources and then decided not to use them, but since the files were still there, they were applied anyway.

my conf is RPI2 with interfaces in IPV4 and IPV6 available
the detection code for piholeIP parameter is not working and result in a blank that is added in the accretiondisc.txt instead of the real IP (resulting in a list of domains only with no sink IP)

in my case result of command : hostname -I is
192.168.1.55 2a01:e35:xxx:xxx:xxx:xxx:xxx:xxx 2a01:e35:yyy:yyy:yyy:yyy:yyy:yyy
hostname -I |xargs is same output

Should probably select only the 1st IP for copy in accretiondisc.txt

Hello,

pi-hole is blocking some domains, I've added the domains to the whitelist.txt file but it doesn't seems to work.

Could someone help me?

/var/log/pihole.log can grow very large, very quickly. In addition to taking up a bunch of space, it also slows down the Web interface as it parses the file to display statistics.

I want to keep the log files so there can eventually be some long-term statistics integrated into the Web interface. So maybe move the log file somewhere else each day and then truncate the log file:

sudo truncate -s 0 /var/log/pihole.log

I actually meant to only parse the log file for today 's stats, but forgot to use the today variable in the code.

What do you think of using Unbound in a split-horizon configuration to serve records pointing to localhost? There's a section of an article at calomel called Dnsspoof or 'Split horizon' with Unbound DNS that may help. That would mean another package though.

This hosts file blocks many legitimate sites, including both the download site for adobe flash and rival adblock lists (it blocks hosts-file.net which is one of your pihole sources.) It also includes lines at the top that give domain names to the hosts 'debian,' and 'android,' which may not be wanted by some people.

If your hosts file happens to be formatted incorrectly, each address gets it's own log line and when you load the Web dashboard, it shows the wrong amount of ads. So your log file will have entries like this:

Nov 29 13:36:20 dnsmasq[5015]: bad address at /etc/pihole/gravity.list line 66877
Nov 29 13:36:20 dnsmasq[5015]: bad address at /etc/pihole/gravity.list line 66878
Nov 29 13:36:20 dnsmasq[5015]: bad address at /etc/pihole/gravity.list line 66879
Nov 29 13:36:20 dnsmasq[5015]: bad address at /etc/pihole/gravity.list line 66880
Nov 29 13:36:20 dnsmasq[5015]: bad address at /etc/pihole/gravity.list line 66881
Nov 29 13:36:20 dnsmasq[5015]: bad address at /etc/pihole/gravity.list line 66882

Each line is counted as an ad since /etc/pihole/gravity.list is found from the query. The fix is probably simple; just adding not bad address into the awk statement.

Hi,

I just learned the latest weekly update blocked me out of (looking at) Google Analytics (analytics.google.com). I should be able to whitelist it by adding a whitelist.txt file in /etc/pi-hole, however I am not sure what format the whitelist should be? Could you give an example?

Thanks! Alex