s0md3v / uro

declutters url lists for crawling/pentesting

uro

Using a URL list for security testing can be painful as there are a lot of URLs that have uninteresting/duplicate content; uro aims to solve that.

It doesn't make any http requests to the URLs and removes:

• incremental urls e.g. /page/1/ and /page/2/
• blog posts and similar human written content e.g. /posts/a-brief-history-of-time
• urls with same path but parameter value difference e.g. /page.php?id=1 and /page.php?id=2
• images, js, css and other "useless" files

Installation

The recommended way to install uro is through pip as follows:

pip3 install uro

Basic Usage

The quickest way to include uro in your workflow is to feed it data through stdin and print it to your terminal.

cat urls.txt | uro

Advanced usage

Reading urls from a file (-i/--input)

uro -i input.txt

Writing urls to a file (-o/--output)

If the file already exists, uro will not overwrite the contents. Otherwise, it will create a new file.

uro -i input.txt -o output.txt

Whitelist (-w/--whitelist)

uro will ignore all other extensions except the ones provided.

uro -w php asp html

Note: Extensionless pages e.g. /books/1 will still be included. To remove them too, use --filter hasext.

Blacklist (-b/--blacklist)

uro will ignore the given extensions.

uro -b jpg png js pdf

Note: uro has a list of "useless" extensions which it removes by default; that list will be overridden by whatever extensions you provide through blacklist option. Extensionless pages e.g. /books/1 will still be included. To remove them too, use --filter hasext.

Filters (-f/--filters)

For granular control, uro supports the following filters:

1. hasparams: only output urls that have query parameters e.g. http://example.com/page.php?id=
2. noparams: only output urls that have no query parameters e.g. http://example.com/page.php
3. hasext: only output urls that have extensions e.g. http://example.com/page.php
4. noext: only output urls that have no extensions e.g. http://example.com/page
5. keepcontent: keep human written content e.g. blogs.
6. keepslash: don't remove trailing slash from urls e.g. http://example.com/page/
7. vuln: only output urls with parameters that are know to be vulnerable. More info.

Example: uro --filters hasexts hasparams

uro's Issues

I get a warning when installing uro with pip

Hello, in older versions of Kalin, I used to write 'pip3 install uro --user' and could directly use Uro. Now, I'm getting a warning as shown in the screenshot on SS and I can't use it. Can you help me with what I need to do?

enhanced filtration

like i want to filter "/A/embed?url=" or "/B/embed?url=" which return similar data
like i want to filter "/A.php" or "/A.php/" which return similar data

Uro error

λ cat newfile222.txt | uro
Traceback (most recent call last):
File "C:\Users\Yaseen\AppData\Local\Programs\Python\Python39\Scripts\uro-script.py", line 33, in
sys.exit(load_entry_point('uro==0.0.1', 'console_scripts', 'uro')())
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\site-packages\uro\uro.py", line 139, in main
if matches_patterns(path):
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\site-packages\uro\uro.py", line 107, in matches_patterns
if re.search(pattern, path):
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\re.py", line 201, in search
return _compile(pattern, flags).search(string)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_parse.py", line 948, in parse
p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_parse.py", line 443, in _parse_sub
itemsappend(_parse(source, state, verbose, nested + 1,
File "c:\users\yaseen\appdata\local\programs\python\python39\lib\sre_parse.py", line 836, in _parse
raise source.error("missing ), unterminated subpattern",
re.error: missing ), unterminated subpattern at position 379
cat: write error: No space left on device

Can you help, it saying space issue, i have alot of space

false positive

hmmm ...

Add support to decode different typees of encoded data

Multiple URL either have parameters with base64 encoding or any other encodings

Output formating

My suggestion is to get the output in json format, to tell if a particular url was decoded, and what encoding was present.

Bad character range P-C at position 31

cat urls.txt | uro

Traceback (most recent call last):
  File "/usr/local/bin/uro", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/uro/uro.py", line 155, in main
    if re.search(pattern, path):
  File "/usr/lib/python3.8/re.py", line 201, in search
    return _compile(pattern, flags).search(string)
  File "/usr/lib/python3.8/re.py", line 304, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.8/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.8/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/lib/python3.8/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/usr/lib/python3.8/sre_parse.py", line 598, in _parse
    raise source.error(msg, len(this) + 1 + len(that))
re.error: bad character range P-C at position 31

Help

How Can I Get The Result In A txt File?

It doesn't delete paths

When we check the paths, we see that 43935989 and 43935976 are used differently.

root@localhost:~# cat urls.txt
https://news.mail.ru/politics/43935976/?social=tw
https://news.mail.ru/politics/43935989/?social=tw

it should delete one of them but it doesn't.

root@localhost:~# cat urls.txt | uro
https://news.mail.ru/politics/43935976/?social=tw
https://news.mail.ru/politics/43935989/?social=tw

ImportError: cannot import name 'SIGPIPE' from 'signal'

D:\uro>uro
Traceback (most recent call last):
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\Scripts\uro-script.py", line 33, in
sys.exit(load_entry_point('uro==0.0.2', 'console_scripts', 'uro')())
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\Scripts\uro-script.py", line 25, in importlib_load_entry_point
return next(matches).load()
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\importlib\metadata.py", line 77, in load
module = import_module(match.group('module'))
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\importlib_init_.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 655, in _load_unlocked
File "", line 618, in _load_backward_compatible
File "", line 259, in load_module
File "C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\site-packages\uro-0.0.2-py3.8.egg\uro\uro.py", line 4, in
ImportError: cannot import name 'SIGPIPE' from 'signal' (C:\Users\umara\AppData\Local\Programs\Python\Python38\lib\signal.py)

install issue

when i run this command (python3 setup.py install) it install but not working. OS=kali

ERROR

i just can't get this to work
have cloned the repo and run the install command, bur when i try "cat file.txt | uro" it dosen't work.
do i have to do any additional commands?
any installation video??:)

error handling

So I added uro to my workflow and after a while I got this error:

Traceback (most recent call last):
  File "/usr/local/bin/uro", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/uro/uro.py", line 139, in main
    if matches_patterns(path):
  File "/usr/local/lib/python3.8/dist-packages/uro/uro.py", line 107, in matches_patterns
    if re.search(pattern, path):
  File "/usr/lib/python3.8/re.py", line 201, in search
    return _compile(pattern, flags).search(string)
  File "/usr/lib/python3.8/re.py", line 304, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib/python3.8/sre_compile.py", line 764, in compile
    p = sre_parse.parse(p, flags)
  File "/usr/lib/python3.8/sre_parse.py", line 948, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "/usr/lib/python3.8/sre_parse.py", line 443, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
  File "/usr/lib/python3.8/sre_parse.py", line 836, in _parse
    raise source.error("missing ), unterminated subpattern",
re.error: missing ), unterminated subpattern at position 369

It is happening to me with different inputs so seems to be something that happens often

installed but not working

Improvement Request

Hi Somdev,

1. I'd like to suggest you add the following extensions to be blacklisted. I have gathered all of these extensions manually and I think It would be nice to omit them:

'svg','img','gif','mp4','flv','ogv','webm','webp','mov','mp3','m4a','m4p','ppt','pptx','pdf','scss','tif','tiff','ttf','otf','woff','woff2','eot','htc','swf','rtf','image'

2. Also, I would like to ask for white-listing and allowing the js extension as there are lots of interesting features/endpoints to be found on them and I don't think if they are considered "useless".

Thanks!

Kind Regards,
HolyBugx

When using uro on subdomains it leaves :// in front?

When using uro on subdomains it leaves :// in front

example:

cat subs.txt | uro

subs.txt example:
site.com
sub.site.com
sub123.site.com

anything without http:// or https:// in front it leaves the :// in front.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 7841: invalid continuation byte

I am using "uro" with other tools to filter duplicate urls, jpegs, etc., and so on. However, after feeding a large wordlist to "uro" with a lot of url paths ending with /678XXXXXXXXXXXXXXXXXXXXXXX, it crashed. Later on, with other normal urls, it no longer works as usual. I installed the uro by pipx-

pipx install uro

Steps to reproduce:-

cat link_results.txt | uro | tee -a output_uro.txt
Traceback (most recent call last):
  File "/usr/local/bin/uro", line 33, in <module>
    sys.exit(load_entry_point('uro==0.0.5', 'console_scripts', 'uro')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/uro-0.0.5-py3.11.egg/uro/uro.py", line 123, in main
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 7841: invalid continuation byte

In the test environment I have -
uro 1.0.0-beta
Python3 Version - 3.11.9
kali 6.8.11-amd64 kernel
pipx version 1.6.0

It might fix if improve or update in the code regarding UnicodeDecode.

URO for POST Request

Can i use URO for decluttering post request to use fetch unique requests using ffuf

Error

Traceback (most recent call last):
File "/usr/local/bin/uro", line 33, in
sys.exit(load_entry_point('uro==0.0.4', 'console_scripts', 'uro')())
File "/usr/local/bin/uro", line 25, in importlib_load_entry_point
return next(matches).load()
StopIteration

broken pipe

I have been encountering this issue:

  File "/usr/local/bin/uro", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/uro/uro.py", line 151, in main
    print(host + path + dict_to_params(param))
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/bin/uro", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/uro/uro.py", line 161, in main
    print(host + path)
BrokenPipeError: [Errno 32] Broken pipe

Any idea why would it be?

PLease solve this

└─# cat params.csv | uro | wc -l
Traceback (most recent call last):
File "/usr/local/bin/uro", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.9/dist-packages/uro/uro.py", line 155, in main
if re.search(pattern, path):
File "/usr/lib/python3.9/re.py", line 201, in search
return _compile(pattern, flags).search(string)
File "/usr/lib/python3.9/re.py", line 304, in _compile
p = sre_compile.compile(pattern, flags)
File "/usr/lib/python3.9/sre_compile.py", line 764, in compile
p = sre_parse.parse(p, flags)
File "/usr/lib/python3.9/sre_parse.py", line 962, in parse
raise source.error("unbalanced parenthesis")
re.error: unbalanced parenthesis at position 68
6547

uro error

cat urls.txt | uro > test

Traceback (most recent call last):
File "/usr/local/bin/uro", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/uro/uro.py", line 123, in main
for line in sys.stdin:
File "/usr/lib/python3.10/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

@s0md3v

ValueError: Invalid IPv6 URL

cat all_urls.txt | uro

Traceback (most recent call last):
File "/usr/local/bin/uro", line 11, in
load_entry_point('uro==0.0.4', 'console_scripts', 'uro')()
File "/usr/local/lib/python3.10/dist-packages/uro-0.0.4-py3.10.egg/uro/uro.py", line 124, in main
File "/usr/lib/python3.10/urllib/parse.py", line 392, in urlparse
splitresult = urlsplit(url, scheme, allow_fragments)
File "/usr/lib/python3.10/urllib/parse.py", line 483, in urlsplit
raise ValueError("Invalid IPv6 URL")
ValueError: Invalid IPv6 URL

Error install uro

suya has the error...
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behavior with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

I've done the steps above but haven't found a bright spot :(

can anyone help me???

Update to new version

Hi

I've installed the old version and how can I update to new version?
After installing the new version how can I know about the version? There isn't any help menu!

Please add more info about the update and help menu in readme file.

Thanks for sharing your code

Can URO work on Windows

tried downloading URO on windows machine, is it possible to use uro on Wndows machine

Unable to install URO

I used pip command but not installed.

It shows :

Requirement already satisfied: uro in ./.local/lib/python3.11/site-packages (1.0.0)

More extension to declutter

Maybe it can be useful to add this extension to the one to declutter, at least, it's what I usually do:

.doc
.docx
.mp3
.mp4
.exe
.tif
.ttf
.woff
.woff2
.ico
.zip

File filtering

Remove "js" file filtering

Not checking similar web content

Hello,
This is not checking similar web content on many links. I will share the screenshot and URLs. Please fix it.

extend filtering

extend ext filter css|jpg|png|gif|svg|ico|woff|woff2|pdf|ttf|otf|jpeg|JPG|js|mp4|mp3|avi|eot

Command 'uro' not found

after installing
pip3 install uro when use this command
cat url.txt | uro it doesn't work as the photo

All of a sudden wont work

Traceback (most recent call last):
File "/usr/bin/uro", line 8, in
sys.exit(main())
^^^^^^
File "/home/greyhound/.local/lib/python3.11/site-packages/uro/uro.py", line 163, in main
process_url(parsed_url)
File "/home/greyhound/.local/lib/python3.11/site-packages/uro/uro.py", line 136, in process_url
new_params = [param for param in params.keys() if param not in params_seen]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/greyhound/.local/lib/python3.11/site-packages/uro/uro.py", line 136, in
new_params = [param for param in params.keys() if param not in params_seen]
^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Ive tried tried setting aliases, changing paths etc wont work..

--filters noexts Not working