Whitelist support about pi-hole

[jacobsalmela]

I have been wanting to figure out a way to do this for a while, as a lot of people request it. However, this single-line command has gotten pretty complex and as it was a pull-request, I don't fully understand it. I see that you are trying to use a whitelist file to somehow remove the entries from the list.

Could you break down each pipe in the command and explain what each one does?

[lailoken]

Well, initially I wanted to separate them into subsequent statements, but I then decided to avoid that because I wanted to limit the amount of writes to the SD card.

All I did was to insert the following into the pipe chain: cat - /usr/local/etc/whitelist.txt | sort | uniq -u

I did this just after a sanitized, uniq-ued, sorted list of addresses were generated, but before the DNS rules were generated.

This addition adds the whitelisted entries to the list ('-' is the list read from stdin), sorts it again, and uses uniq with the '-u' flag to remove all entries with duplicates (thus the whitelisted ones).

There are 2 problems with my current implementation:

1. If the whitelist file does not exist, it will fail. This can be handled better with a check-and-read statement.
2. The bigger problem is that if the addresses I have in the whitelist get removed from the blacklist, then my addition will actually add them again! There is an easy fix for this as well, use:

cat - /usr/local/etc/whitelist.txt /usr/local/etc/whitelist.txt | sort | uniq -u

(adding whitelist entries twice will ensure they are always removed, I'll update my original comment.)

No. 2) above can be done better with regular expressions/awk scripts, but I'm lazy... :)

[lailoken]

ps: my whitelist.txt contains an address-per-line like so:

imgs.xkcd.com
c.xkcd.com

[jacobsalmela]

Interesting. I'll look at your code more deeply and see what I can figure out.

Why does the whitelist need to be in there twice? I also don't really get what the - is for.

The error check should be fairly simple:

if [[ -f /usr/local/etc/whitelist.txt ]];then
command to remove
else
echo "File does not exist"
fi

[lailoken]

Yup, I forgot to mention (added the info too late) : In cat -, the - reads from stdin (thus from the pipe).

cat - whitelist.txt whitelist.txt , will basically then combine the list from the pipe, with two copies from the whitelist.

And the reason the list is added twice, is for the eventuality that the address I whitelisted is removed from the blacklist site, then it prevents that code from actually adding it again.

This has to do with my 'hack' to get it working quickly: uniq with the -u parameter basically only returns the lines which it did not find duplicated.

[jacobsalmela]

What about something like this?
~$ cat adList.conf

0-29.com
0-2u.com
0-days.net
0-free-porn-0.com
0-pdf.com
0.asdzc123.info
0.channel15.facebook.com
0.datacollector.coin.scribol.com
0.doctoraedom.ru
0.doctoralduz.ru

~$ cat whitelist.txt

0-pdf.com
0.datacollector.coin.scribol.com

~$ grep -v -x -f whitelist.txt adList.conf

0-29.com
0-2u.com
0-days.net
0-free-porn-0.com
0.asdzc123.info
0.channel15.facebook.com
0.doctoraedom.ru
0.doctoralduz.ru

I'm not quite sure how to implement it into the command yet.

[lailoken]

This is indeed simpler!

So instead of:
| cat - /usr/local/etc/whitelist.txt /usr/local/etc/whitelist.txt | sort | uniq -u |

Thus do the following:
| grep -v -x -f whitelist.txt |

[jacobsalmela]

Cool! I will work a little more logic in to check for the file and then update the script.

[jacobsalmela]

Just add a whitelist.txt file to your home folder with one domain per line.

b5cf6de