GithubHelp home page GithubHelp logo

urlame's Introduction

urlame

This tool can reduce a list of URLs in a way which should be useful for pentesting / bug bounty.
E.g., when searching interesting URLs in the output of tools like waymore, this can do some initial filtering.

image

urlame aims to print one URL per feature of the website in addition to blocking known lame URLs.
This is done by converting a URL into a pattern and matching that against the patterns seen before.

Things urlame considers lame

As a first step, urlame will filter out:

  • lame directories like /docs
  • files with lame extentensions like .png
  • URLs that look like blog posts
  • user profile/referral pages like /user/FooBar

This tool also ignores query values, so that only if a new parameter appears on a specific endpoint, the URL will be listed.
This means once /foo?id=bar was seen, /foo?id=baz will not be printed.
Certain URL query parameters are ignored completely, so that /foo and /foo?utm_source=twitter are considered equal.

It further can detect some patterns in parts of URLs which are ignored when comparing URLs.

  • language codes
  • numeric IDs
  • hashes
  • UUIDs

This means that /en-US/upload/item/1 and /de-DE/upload/item/5 are considered equal, so only the first will be printed.

Usage

If you don't have Go installed read this.

# installation
go install github.com/wfinn/urlame@latest
# basic usage
urlame < many_urls.txt > less_urls.txt
# practical example
waymore example.org | tee all_urls.txt | urlame > filtered_urls.txt

If you have ideas for more stuff to filter out or find a bug, let me know.

Inspired by uro

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.