Secret attackers??? about blocklist-ipsets

[szepeviktor]

@ktsaou Could you please help to explain this?

These are the blacklists I am using. Spamhaus DNS based blacklist adds 102 to your hits.

      3 bl.de-1h
      3 stopforumspam
      5 httpbl
      7 openbl
     30 cleantalk
     68 greensnow
    102 spamhaus
    209 bl.de

[ktsaou]

What is szerver4?

Generally you cannot grep these files. This is why I developed iprange.
Check this example:

file A:

1.2.3.4

file B:

1.2.0.0/16

You cannot grep them. They do not match.

You should install the development version of FireHOL from github. When you install it, a command line called iprange is installed.

Then to find the common IPs in 2 sets you will need to run:

iprange --common fileA fileB

This will print the common IPs/subnets to both files.

run iprange -h for full usage.

[szepeviktor]

Thank you for your answer.

I've used grep to search in ipsets, and grepcidr to search in netsets. Now I've realised the grepcidr is faster even for single addresses.

szerver4 is the file with the fail2ban-ned IP-s of the previous days.

My aim to reduce fail2ban notification emails during botnet attacks.

You can see in the first comment that I have non-zero results.
Could you help me what to do? Would it help if I send you the IP list?

[ktsaou]

Sorry, I didn't really read the second line.

Yes please do so. Give me a link to download them.

[ktsaou]

or you can just paste them here... they are just 803 IPs
I will tell you which sets its matches among all of them...

[szepeviktor]

OK. Here they are.

(there were IP-s here)

[szepeviktor]

I was not able to go above 48%.

[ktsaou]

ok, here they are:

szervers4:
totals: 803 lines read, 803 distinct IP ranges found, 1 CIDR prefixes, 803 CIDRs printed, 803 unique IPs

Here are what it matches by executing:

iprange --header --compare-first /tmp/szerver4 *.{ip,net}set |\
   grep -v ",0$" |\
   sort -u |\
   tr "," "|"

name	entries	unique_ips	common_ips
alienvault_reputation.ipset	196728	196728	8
bi_any_2_1d.ipset	842	842	4
bi_any_2_30d.ipset	6454	6454	15
bi_any_2_7d.ipset	2597	2597	13
bi_ftp_2_30d.ipset	265	265	1
bi_http_2_30d.ipset	86	86	6
bi_ssh_2_30d.ipset	4942	4942	9
bitcoin_nodes_30d.ipset	47382	47382	1
blocklist_de_apache.ipset	19566	19566	239
blocklist_de_bots.ipset	2604	2604	3
blocklist_de_bruteforce.ipset	8247	8247	238
blocklist_de.ipset	29025	29025	244
blocklist_de_mail.ipset	15446	15446	1
blocklist_de_ssh.ipset	1852	1852	1
blocklist_de_strongips.ipset	141	141	1
blocklist_net_ua.ipset	17622	17622	28
bm_tor.ipset	6420	6420	48
botscout_1d.ipset	1278	1278	34
botscout_30d.ipset	21558	21558	54
botscout_7d.ipset	6318	6318	51
botscout.ipset	51	51	1
cruzit_web_attacks.ipset	4739	4739	3
cybercrime.ipset	5353	5353	12
darklist_de.netset	738	130392	1
dm_tor.ipset	6406	6406	48
dragon_http.netset	950	294144	3
dronebl_anonymizers.netset	167241	174137	8
dronebl_worms_bots.netset	23639	25308	4
dshield_30d.netset	4181	1096704	7
dshield_7d.netset	1514	394240	1
et_tor.ipset	6290	6290	49
firehol_anonymous.netset	27094	85454	50
firehol_level2.netset	28902	137708	309
firehol_level3.netset	100526	10992123	70
firehol_proxies.netset	20739	23787	33
gofferje_sip.netset	507	1094315	5
greensnow.ipset	11834	11834	105
iblocklist_ads.netset	3314	888645	4
iblocklist_bogons.netset	2699	666661539	23
iblocklist_cruzit_web_attacks.netset	4711	4733	3
iblocklist_edu.netset	40792	227983904	9
iblocklist_hijacked.netset	535	9177856	1
iblocklist_isp_att.netset	35	55845128	2
iblocklist_isp_charter.netset	21	6138112	6
iblocklist_isp_comcast.netset	33	45121536	6
iblocklist_isp_twc.netset	56	15015936	3
iblocklist_isp_verizon.netset	22	18087936	4
iblocklist_level1.netset	218306	764928098	16
iblocklist_level2.netset	72950	348710251	10
iblocklist_level3.netset	17812	139104927	34
iblocklist_onion_router.netset	6174	6238	48
iblocklist_org_blizzard.netset	7	16794627	1
iblocklist_pedophiles.netset	22798	872217	6
iblocklist_rangetest.netset	515	4346058	8
iblocklist_spider.netset	734	860168	1
iblocklist_spyware.netset	3297	339021	1
iw_spamlist.ipset	3601	3601	1
lashback_ubl.ipset	371205	371205	12
maxmind_proxy_fraud.ipset	446	446	32
myip.ipset	2672	2672	2
nixspam.ipset	22953	22953	1
openbl_180d.ipset	14242	14242	4
openbl_30d.ipset	2480	2480	2
openbl_360d.ipset	28764	28764	7
openbl_60d.ipset	4830	4830	3
openbl_7d.ipset	780	780	2
openbl_90d.ipset	7458	7458	3
openbl_all.ipset	107331	107331	8
php_commenters_1d.ipset	91	91	3
php_commenters_30d.ipset	807	807	18
php_commenters_7d.ipset	281	281	7
php_commenters.ipset	50	50	2
php_dictionary_30d.ipset	1240	1240	1
php_harvesters_1d.ipset	77	77	4
php_harvesters_30d.ipset	631	631	4
php_harvesters_7d.ipset	199	199	4
php_harvesters.ipset	50	50	4
php_spammers_30d.ipset	1208	1208	1
proxylists_1d.ipset	6691	6691	1
proxylists_30d.ipset	17360	17360	1
proxylists_7d.ipset	9915	9915	1
proxylists.ipset	4730	4730	1
proxyrss_1d.ipset	6421	6421	1
proxyrss_30d.ipset	16888	16888	1
proxyrss_7d.ipset	9557	9557	1
proxyrss.ipset	4505	4505	1
proxyspy_1d.ipset	1213	1213	1
proxyspy_30d.ipset	6382	6382	1
proxyspy_7d.ipset	2825	2825	1
pushing_inertia_blocklist.netset	734	37908616	43
ri_web_proxies_30d.ipset	4258	4258	1
ri_web_proxies_7d.ipset	1316	1316	1
sblam.ipset	11730	11730	46
snort_ipfilter.ipset	9508	9508	44
sorbs_dul.netset	533752	351501549	132
sorbs_new_spam.netset	12382	12516	1
sorbs_noserver.netset	11894	21181534	22
sorbs_recent_spam.netset	41401	42458	1
sorbs_web.netset	5284892	5660056	14
sslbl_aggressive.ipset	1044	1044	1
stopforumspam_180d.ipset	415441	415441	67
stopforumspam_1d.ipset	5279	5279	45
stopforumspam_30d.ipset	78928	78928	58
stopforumspam_365d.ipset	912590	912590	70
stopforumspam_7d.ipset	24311	24311	56
stopforumspam_90d.ipset	220782	220782	63
stopforumspam.ipset	220823	220823	64
stopforumspam_toxic.netset	79	551511	1
talosintel_ipfilter.ipset	9524	9524	44
tor_exits_1d.ipset	1067	1067	49
tor_exits_30d.ipset	3018	3018	50
tor_exits_7d.ipset	1482	1482	49
tor_exits.ipset	973	973	48

These are the countries they match:

name	entries	unique_ips	common_ips
anonymous.netset	143	55381	3
continent_af.netset	1760	82187639	46
continent_as.netset	15981	871795336	255
continent_eu.netset	31370	753642059	333
continent_na.netset	16719	1733361597	106
continent_oc.netset	6498	57408602	15
continent_sa.netset	2750	146404832	51
country_ae.netset	179	3783522	11
country_af.netset	78	172963	1
country_al.netset	112	383252	2
country_am.netset	107	616880	1
country_ar.netset	1018	19018000	4
country_at.netset	1355	11811784	1
country_au.netset	5622	49652472	12
country_az.netset	127	753680	2
country_bb.netset	52	189073	1
country_bd.netset	410	1106184	5
country_be.netset	1241	28868912	2
country_bg.netset	602	4625364	4
country_bh.netset	71	476156	1
country_br.netset	1576	82387918	39
country_by.netset	119	1902617	1
country_ca.netset	6767	80309147	15
country_ch.netset	2030	21035918	2
country_ci.netset	30	198272	2
country_cl.netset	655	10255633	4
country_cn.netset	4089	335945582	8
country_co.netset	498	17433808	2
country_cy.netset	236	1091790	2
country_cz.netset	1144	9194655	7
country_de.netset	7462	122534581	21
country_dk.netset	1030	13250121	2
country_dz.netset	61	3788345	12
country_ee.netset	245	1405273	1
country_eg.netset	183	17346708	3
country_es.netset	1962	30743158	10
country_fi.netset	875	13740673	1
country_fj.netset	33	146736	1
country_fr.netset	6899	84855601	26
country_gb.netset	9657	127030038	36
country_ge.netset	163	1201743	3
country_gh.netset	108	826013	2
country_gr.netset	408	6309290	11
country_hk.netset	1952	11961186	3
country_hn.netset	231	518114	2
country_hr.netset	255	2415262	2
country_hu.netset	454	5927383	5
country_id.netset	1304	19099320	3
country_ie.netset	1025	8378688	3
country_il.netset	505	7885896	26
country_in.netset	2902	38137351	32
country_iq.netset	143	679476	5
country_ir.netset	864	11650123	4
country_it.netset	2748	53977942	20
country_je.netset	45	68596	2
country_jm.netset	85	274702	4
country_jo.netset	129	680340	4
country_jp.netset	2847	204400580	7
country_kr.netset	940	112426449	18
country_kz.netset	255	2968588	3
country_lt.netset	330	2608455	10
country_lu.netset	245	1434431	2
country_lv.netset	374	1859829	6
country_ly.netset	37	335364	1
country_ma.netset	52	6744960	5
country_md.netset	350	1360520	3
country_me.netset	41	221456	3
country_mk.netset	88	687632	5
country_mn.netset	68	214656	4
country_mu.netset	262	2051837	1
country_mv.netset	15	58368	1
country_mx.netset	662	28905817	4
country_my.netset	526	6688718	7
country_ng.netset	427	2728090	3
country_nl.netset	5111	50562294	19
country_no.netset	1058	16146767	6
country_np.netset	83	503425	4
country_nz.netset	836	7144953	2
country_om.netset	44	905236	1
country_ph.netset	519	5506676	33
country_pk.netset	287	5278634	23
country_pl.netset	3145	20759970	8
country_ps.netset	103	599300	1
country_pt.netset	443	6457274	7
country_py.netset	69	1099552	1
country_qa.netset	52	831500	3
country_ro.netset	2270	9830672	35
country_rs.netset	304	2285074	20
country_ru.netset	6809	46257206	30
country_sa.netset	378	8693764	15
country_se.netset	2290	30553879	9
country_sg.netset	1340	7383813	7
country_si.netset	438	2805027	7
country_sk.netset	327	2799418	2
country_sr.netset	17	87178	1
country_th.netset	472	9027650	2
country_tn.netset	20	4964736	4
country_tr.netset	769	16457939	8
country_tt.netset	47	526488	1
country_tw.netset	495	35509252	1
country_ua.netset	2858	11855803	9
country_ug.netset	62	313756	1
country_us.netset	20664	1620780943	79
country_uz.netset	69	240128	1
country_vc.netset	61	35837	1
country_vn.netset	367	15756001	7
country_za.netset	1005	28093655	9
country_zm.netset	85	1175992	2
country_zw.netset	61	144896	1

The IPs you posted, in how many days did you collect them?

[ktsaou]

The IPs that are matched by all my lists are:

iprange --merge *.{ip,net}set |\
   iprange --header --compare /tmp/szerver4 - |\
   tr "," "|"

name1	name2	ips1	ips2	combined_ips	common_ips
stdin	/tmp/szerver4	2654632870	803	2654633178	495

[szepeviktor]

Oct 18 - Oct 21 (3,5 days)
My WordPress WAF is very strict.

[ktsaou]

Well, it seems there are 308 IPs that are not matched by any of the lists I monitor...

[szepeviktor]

Thank very much you for your work!!
I still believe that my WAF irritates botnets.

[szepeviktor]

I appreciate your help.