projectdiscovery / wappalyzergo

A high performance go implementation of Wappalyzer Technology Detection Library

Wappalyzergo

A high performance port of the Wappalyzer Technology Detection Library to Go. Inspired by Webanalyze.

Uses data from https://github.com/AliasIO/wappalyzer

Features

• Very simple and easy to use, with clean codebase.
• Normalized regexes + auto-updating database of wappalyzer fingerprints.
• Optimized for performance: parsing HTML manually for best speed.

Using go install

go install -v github.com/projectdiscovery/wappalyzergo/cmd/update-fingerprints@latest

After this command wappalyzergo library source will be in your current go.mod.

Example

Usage Example:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"

	wappalyzer "github.com/projectdiscovery/wappalyzergo"
)

func main() {
	resp, err := http.DefaultClient.Get("https://www.hackerone.com")
	if err != nil {
		log.Fatal(err)
	}
	data, _ := io.ReadAll(resp.Body) // Ignoring error for example

	wappalyzerClient, err := wappalyzer.New()
	fingerprints := wappalyzerClient.Fingerprint(resp.Header, data)
	fmt.Printf("%v\n", fingerprints)

	// Output: map[Acquia Cloud Platform:{} Amazon EC2:{} Apache:{} Cloudflare:{} Drupal:{} PHP:{} Percona:{} React:{} Varnish:{}]
}

wappalyzergo's Issues

How does updating fingerprint works

Hi Team,

As always another major and interesting opensource project from you all 🔥

One of the first thing that I noticed that https://github.com/rverton/webanalyze downloads the fingerprints from the wappalyzer repo and save them locally and also has a update flag which can be used to update the same.

But looks like this project is hard coding them in https://github.com/projectdiscovery/wappalyzergo/blob/master/fingerprints_data.go which is weird considering you have to update the code every time their is an update in the wappalyzer fingerprint which is quite frequent.

Also the original file https://github.com/AliasIO/wappalyzer/blob/master/src/technologies.json is of 22181 lines where as even if we beautify https://github.com/projectdiscovery/wappalyzergo/blob/master/fingerprints_data.go its of 10861 lines only. Would love to understand why is this difference, are you ignoring non useful tags from the list ?

I did see this https://github.com/projectdiscovery/wappalyzergo/blob/master/cmd/update-fingerprints/main.go But how does this works if am using this in my code like below, does it tries to update the fingerprint every time wappalyzer.New() is being called ?
Can we manually invoke this update part using something like wappalyzer.update() only once before reusing the same wappalyzerClient.

      for _, url range urls{
	resp, err := http.DefaultClient.Get(url)
	if err != nil {
		log.Fatal(err)
	}
	data, _ := ioutil.ReadAll(resp.Body) // Ignoring error for example

	wappalyzerClient, err := wappalyzer.New()
	fingerprints := wappalyzerClient.Fingerprint(resp.Header, data)
	fmt.Printf("%v\n", fingerprints)
       }

--
Regards,
@bugbaba

Issue in cookie based tech detection

Sorry for my bad english, using google translate.

defines only if the fingerprint is immediately after Set-Cookie:
eg (Set-Cookie:[laravel_session=)
but if somewhere in the middle, such as (Set-Cookie:[XSRF-TOKEN=; expires=, ...; ...; path=/ laravel_session=eyJ*;)
then no longer defines

example domains:
hit4central.com
shcherbyna-advocat.com
amiguiec.xyz
hiom.in
puncbimsi.cf
suzuki.com.pl

A third of the technologies listed in fingerprints_data.json will never be detected

A third of the technologies listed in fingerprints_data.json will never be detected, as the only keys listed there are the technology name and description
examples:
"AddShoppers":{"description": "AddShoppers is the ...
"ECharts":{"description": "ECharts is an open-source...
...
And there are 884 similar technologies, which only add weight to file fingerprints_data.json for nothing, because from original file keys are missing: scriptSrc, dom

P.S.: Apologies for my bad English, I use a translator

Add headless based wappalyzer implementation

Add injectable javascript agent for technologies detection

An always up-to-date javascript injectable library version should be prepared and kept up to date. The library should be injected via headless into existing browsers' contexts and collect enriched info within the JS engine for a specific domain open in a browser tab. This is similar to the behavior of the official wappalyzer extension https://chrome.google.com/webstore/detail/wappalyzer-technology-pro

Underreporting bug

package main

import (
	"fmt"
	wappalyzer "github.com/projectdiscovery/wappalyzergo"
)

func main() {
	wappalyzerClient, _ := wappalyzer.New()
	fingerprints := wappalyzerClient.Fingerprint(map[string][]string{
		"Set-Cookie": []string{"path=/;  jsessionid=111; path=/, jsessionid=111;"},
	}, []byte(""))
	fingerprints1 := wappalyzerClient.Fingerprint(map[string][]string{
		"Set-Cookie": []string{"jsessionid=111; path=/;"},
	}, []byte(""))

	fmt.Println(fingerprints)
	fmt.Println(fingerprints1)
}

output:

map[]
map[Java:{}]

If the value changes position, the test will fail

SIGSEGV: segmentation violation code crash (mac)

51pwn@123 update-fingerprints $ go build main.go 
51pwn@123 update-fingerprints $ ./main 
2022/06/24 13:24:01 Read fingerprints from the server
2022/06/24 13:24:01 Starting normalizing of 3084 fingerprints...
2022/06/24 13:24:01 Got 3084 valid fingerprints
unexpected fault address 0xb01dfacedebac1e
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x105f71f]

goroutine 1 [running]:
runtime.throw({0x12c771b?, 0x1?})
	/usr/local/Cellar/go/1.18.2/libexec/src/runtime/panic.go:992 +0x71 fp=0xc0001b5628 sp=0xc0001b55f8 pc=0x1033871
runtime.sigpanic()
	/usr/local/Cellar/go/1.18.2/libexec/src/runtime/signal_unix.go:825 +0x305 fp=0xc0001b5678 sp=0xc0001b5628 pc=0x1048d25
aeshashbody()
	/usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:1343 +0x39f fp=0xc0001b5680 sp=0xc0001b5678 pc=0x105f71f
runtime.mapiternext(0xc0000e4740)
	/usr/local/Cellar/go/1.18.2/libexec/src/runtime/map.go:934 +0x2cb fp=0xc0001b56f0 sp=0xc0001b5680 pc=0x101084b
runtime.mapiterinit(0x0?, 0x1?, 0x1b?)
	/usr/local/Cellar/go/1.18.2/libexec/src/runtime/map.go:861 +0x228 fp=0xc0001b5710 sp=0xc0001b56f0 pc=0x1010528
reflect.mapiterinit(0x2?, 0x1273fe0?, 0x100b195?)
	/usr/local/Cellar/go/1.18.2/libexec/src/runtime/map.go:1373 +0x19 fp=0xc0001b5738 sp=0xc0001b5710 pc=0x105c759
github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
	/Users/51pwn/go/pkg/mod/github.com/modern-go/[email protected]/unsafe_map.go:112
github.com/json-iterator/go.(*mapEncoder).Encode(0xc0002f1050, 0xc0001a85a8, 0xc000148060)
	/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_map.go:257 +0x21c fp=0xc0001b57f0 sp=0xc0001b5738 pc=0x123917c
github.com/json-iterator/go.(*structFieldEncoder).Encode(0xc0002f10b0, 0x125c0ce?, 0xc000148060)
	/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:110 +0x56 fp=0xc0001b5868 sp=0xc0001b57f0 pc=0x1247176
github.com/json-iterator/go.(*structEncoder).Encode(0xc0002f10e0, 0x0?, 0xc000148060)
	/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_struct_encoder.go:158 +0x765 fp=0xc0001b5950 sp=0xc0001b5868 pc=0x1247b85
github.com/json-iterator/go.(*OptionalEncoder).Encode(0xc0001ae960?, 0x0?, 0x0?)
	/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect_optional.go:70 +0xa4 fp=0xc0001b59a0 sp=0xc0001b5950 pc=0x123ed84
github.com/json-iterator/go.(*onePtrEncoder).Encode(0xc00060e940, 0xc0001a85a8, 0xc0002f0120?)
	/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:219 +0x82 fp=0xc0001b59d8 sp=0xc0001b59a0 pc=0x1232082
github.com/json-iterator/go.(*Stream).WriteVal(0xc000148060, {0x1267fa0, 0xc0001a85a8})
	/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/reflect.go:98 +0x158 fp=0xc0001b5a48 sp=0xc0001b59d8 pc=0x1231398
github.com/json-iterator/go.(*frozenConfig).Marshal(0xc0001ae960, {0x1267fa0, 0xc0001a85a8})
	/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/config.go:299 +0xc9 fp=0xc0001b5ae0 sp=0xc0001b5a48 pc=0x1229ba9
github.com/json-iterator/go.Marshal(...)
	/Users/51pwn/go/pkg/mod/github.com/json-iterator/[email protected]/adapter.go:34
main.main()
	/Users/51pwn/MyWork/wappalyzergo/cmd/update-fingerprints/main.go:96 +0x2ca fp=0xc0001b5f80 sp=0xc0001b5ae0 pc=0x125004a
runtime.main()
	/usr/local/Cellar/go/1.18.2/libexec/src/runtime/proc.go:250 +0x212 fp=0xc0001b5fe0 sp=0xc0001b5f80 pc=0x1035f52
runtime.goexit()
	/usr/local/Cellar/go/1.18.2/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc0001b5fe8 sp=0xc0001b5fe0 pc=0x1062341

goroutine 7 [IO wait]:
internal/poll.runtime_pollWait(0x25c3120, 0x72)
	/usr/local/Cellar/go/1.18.2/libexec/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000248300?, 0xc00050e000?, 0x0)
	/usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000248300, {0xc00050e000, 0xa954, 0xa954})
	/usr/local/Cellar/go/1.18.2/libexec/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc000248300, {0xc00050e000?, 0xc000076380?, 0xc00050e005?})
	/usr/local/Cellar/go/1.18.2/libexec/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc000010010, {0xc00050e000?, 0x243ffff?, 0x2245700?})
	/usr/local/Cellar/go/1.18.2/libexec/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc00016e720, {0xc00050e000?, 0x0?, 0x500?})
	/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:784 +0x3d
bytes.(*Buffer).ReadFrom(0xc00010e278, {0x133e9a0, 0xc00016e720})
	/usr/local/Cellar/go/1.18.2/libexec/src/bytes/buffer.go:204 +0x98
crypto/tls.(*Conn).readFromUntil(0xc00010e000, {0x25fd008?, 0xc000010010}, 0x203000?)
	/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:806 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc00010e000, 0x0)
	/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:613 +0x116
crypto/tls.(*Conn).readRecord(...)
	/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:581
crypto/tls.(*Conn).Read(0xc00010e000, {0xc00014f000, 0x1000, 0x11e5d60?})
	/usr/local/Cellar/go/1.18.2/libexec/src/crypto/tls/conn.go:1284 +0x16f
bufio.(*Reader).Read(0xc0001484e0, {0xc0001364a0, 0x9, 0x11f36e2?})
	/usr/local/Cellar/go/1.18.2/libexec/src/bufio/bufio.go:236 +0x1b4
io.ReadAtLeast({0x133e8c0, 0xc0001484e0}, {0xc0001364a0, 0x9, 0x9}, 0x9)
	/usr/local/Cellar/go/1.18.2/libexec/src/io/io.go:331 +0x9a
io.ReadFull(...)
	/usr/local/Cellar/go/1.18.2/libexec/src/io/io.go:350
net/http.http2readFrameHeader({0xc0001364a0?, 0x9?, 0xc0002160f0?}, {0x133e8c0?, 0xc0001484e0?})
	/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:1566 +0x6e
net/http.(*http2Framer).ReadFrame(0xc000136460)
	/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:1830 +0x95
net/http.(*http2clientConnReadLoop).run(0xc0001b9f98)
	/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:8815 +0x130
net/http.(*http2ClientConn).readLoop(0xc00010c180)
	/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:8711 +0x6f
created by net/http.(*http2Transport).newClientConn
	/usr/local/Cellar/go/1.18.2/libexec/src/net/http/h2_bundle.go:7439 +0xa65

Broken Package - GH Update Action

Description

The GitHub update action points to the wrong file fingerprints_data.go instead of fingerprints_data.json:

• fingerprints_data.go => fingerprints_data.json
• Restore fingerprints_data.go with embed
• Release new version (cc @ehsandeep )

Add service's website information to wappalyzer fingerprint database

Wappalyzer is Now Closed Source

Sadly, Wappalyzer has been removed from github due to some bitter feelings. The links to the signature files now result in 404s.

However, it is still possible to pull the signatures from the Chrome extension. This gets you a crx file, which similar to a jar or apk is just a glorified zip. The signatures can be downloaded and extracted from the following URL:

https://clients2.google.com/service/update2/crx?response=redirect&prodversion=119.0.6045.199&acceptformat=crx2,crx3&x=id%3Dgppongmhjkpfnbhagpmjfkannfbllamg%26installsource%3Dondemand%26uc

Change in original technologies.json file structure, Will break wappalyzergo

Hello Team,

As the technologies.json file was growing the wappalyzer team had decide to break it into multiple files based on starting character In this commit https://github.com/AliasIO/wappalyzer/commit/2401457a994f40f23140b62e406590bdaa0476a2

We can see the new json files here https://github.com/AliasIO/wappalyzer/tree/master/src/technologies and as this project depends on technologies.json at https://github.com/projectdiscovery/wappalyzergo/blob/master/cmd/update-fingerprints/main.go#L20 it will return 404 error and break the process.

So now we need to download all thee files and then merge them together, before using them.

--
Regards,
@bugbaba

update fingerprints_data.json problem when use wappalyzergo as library

hi, i use wappalyzergo as library，the fingerprints_data.json file is bound to the repo code. When I update the json file, I need to update the whole golang project, I think it would be nice to have a function on the Wappalyze object to define fingerprints, In this way, when used as a function library, it avoids constantly updating the entire project code in order to update the data file

// now I use some tricks to solve this problem
// the wappalyzerFileBytes define from outter resource, such as cloud object storage service

//go:linkname compileFingerprint github.com/projectdiscovery/wappalyzergo.compileFingerprint
func compileFingerprint(fingerprint *wappalyzer.Fingerprint) *wappalyzer.CompiledFingerprint

type WappalyzeWithUnsafe struct {
	fingerprints unsafe.Pointer
}

func NewWappalyzerClient() *wappalyzer.Wappalyze {
	EnsureWappalyzerFile()

	wappalyzerClient := new(wappalyzer.Wappalyze)
	wappalyzerClientCompiledFingerprints := &wappalyzer.CompiledFingerprints{
		Apps: make(map[string]*wappalyzer.CompiledFingerprint),
	}
	var fingerprintsStruct wappalyzer.Fingerprints
	json.Unmarshal([]byte(wappalyzerFileBytes), &fingerprintsStruct)

	for i, fingerprint := range fingerprintsStruct.Apps {
		wappalyzerClientCompiledFingerprints.Apps[i] = compileFingerprint(fingerprint)
	}

	wappalyzeUnsafe := (*WappalyzeWithUnsafe)(unsafe.Pointer(wappalyzerClient))
	wappalyzeUnsafe.fingerprints = unsafe.Pointer(wappalyzerClientCompiledFingerprints)

	return wappalyzerClient
}

Add version detection too for wappalyzer fingerprint database

• Include version information
• Include website associted with tech

Original Wappalyzer project not open source anymore - Fingerprints failed to download

Hi all!

Pretty recently, the Wappalyzer original project has been made private (see here).

Therefore, the fingerprints fail to download from the original repository.

I saw some discussion here: https://news.ycombinator.com/item?id=37236746

And there are several repositories with the latest snapshot.

Should we consider to start maintaining a list of fingerprints inside this same repository using the original one as a baseline?

I just created a repository for myself that I would love to try to maintain with the community if needed: https://github.com/BBerastegui/wappalyzer-fingerprints

From now we can pull from there.

Cheers!

import categories from upstream

Would you be open to a PR that imports https://github.com/wappalyzer/wappalyzer/blob/master/src/categories.json on a cron, similar to how technologies are pulled weekly, to make category data available?

Add More GitHub actions

Description

Add the following GitHub actions

• CodeQL
• Sonarcloud
• Lint
• DependantBot

sort technologies.json

I'm not sure if the upstream json is sorted randomly or if it is happening during ingestion, but #38 didn't quite solve the problem of making diffs usable.

• #38

Replace huge json dataset with folder-tree structured technologies dataset and readable markup language

The current wappalyzer dataset is kind of unreadable due to size and native json intrinsic unreadability which grows linearly with the size of json. It can be a good occasion to investigate a new implementation and structure the signatures in a folder tree fashion, with a more readable format such as yaml (potentially nuclei templates covering technologies could be reused).

All the signatures must respect the following requirements:

• They are based only on passive detection (no request modifiers or additional requests)
• They should be idempotent, multiple runs should yield the same result