twin / go-away

Library for detecting profanities in Go

Hey, I'm TwiN! 👋

Just another person trying to embrace their passion prevent the vicissitudes of life from taking away their passion.

Currently building my first SaaS during my free time: gatus.io

📝 Latest Blog Articles

• Securing Gatus with OIDC using Auth0
• Mocking HTTP requests in Go
• Bit Fields: Squishing Booleans Using Bitwise Operations
• Writing Good Documentation
• Shit Programmers Say: “Just Scale Up”
• Performance, Benchmarking and Allocations in Go
• What Programming Means to Me
• Go Concurrency: GoRoutines, Worker Pools and Throttling Made Simple

📊 Statistics

go-away's Issues

Collapse repeated characters (possibly using regex)

The program

package main

import (
	"fmt"
	"github.com/TwinProduction/go-away"
)

func main() {
	test("shit")
	test("shiit")
	test("shiiiiiit")
}

func test(word string) {
	fmt.Printf("\"%s\" profane? %t\n", word, goaway.IsProfane(word))
}

prints

"shit" profane? true
"shiit" profane? false
"shiiiiiit" profane? false

I suggest you include a list of regexes to match cases such as this.

An alternative would be to collapse all repeated characters before matching, although this would have an issue with any profanities that contain repeated characters that are no longer profane once collapsed.

Filter only the exact words

Describe the bug

We need to filter out only the exact words provided in the profanities list.
Eg: fr -> con is a profane word. It should be filter if it is an exact word. 'contactes' should not be filtered.

Is there a way can do this? I don't want to use falsepositvies because we are maintaining multiple languages.

Please suggest

What do you see?

No response

What do you expect to see?

No response

List the steps that must be taken to reproduce this issue

We need to filter out only the exact words provided in the profanities list.
Eg: fr -> con is a profane word. It should be filter if it is an exact word. 'contactes' should not be filtered.

Is there a way can do this? I don't want to use falsepositvies because we are maintaining multiple languages.

Please suggest

Version

No response

Additional information

No response

Duplicate false positive causes panic

https://github.com/TwiN/go-away/blob/master/goaway.go#L178

runtime error: slice bounds out of range [457:451]

"Scunthorpe" is detected as profane

Scunthorpe is a town in England and is well known for causing issues with profanity filters.

This might be an acceptable false positive but I thought it's worth mentioning!

Edit: I just realised you have a list of false positives and scunthorpe is in there, it's just spelt incorrectly (with 2 t's instead of 1)

Strange cases of profanity

[PROFANE] are you open for business on saturday
[PROFANE] hello. I have no hot water
[PROFANE] Why do I not have hot water
[PROFANE] my hot water is not working
[PROFANE] what type of HVAC systems exist
[PROFANE] who has stephanie worked for?
[PROFANE] what comapnies has she worked for?
[PROFANE] is sun devil in phoenix?
[PROFANE] What was Stanley Connor doing?
[PROFANE] fuck you
[PROFANE] you're a dumb bitch
[PROFANE] do you want to have sex with me
[PROFANE] no bitch
[PROFANE] fuck off
[PROFANE] bitch
[PROFANE] bitch please
[PROFANE] fuck you!
[NOT] you suck
[NOT] i hate you

Remove word from profanity list

A Vagina is a :

the elastic, muscular part of the female genital tract. In humans, it extends from the vulva to the cervix. The outer vaginal opening is normally partly covered by a membrane called the hymen. At the deep end, the cervix (neck of the uterus) bulges into the vagina. The vagina allows for sexual intercourse and birth. It also channels menstrual flow (menses), which occurs in humans and closely related primates as part of the monthly menstrual cycle.

NOT A PROFANITY.

Support profanity levels

Add support for multiple level of profanity detection.

e.g.

func IdProfane(s string, level int) bool {
    // ...
}

func IsProfane(s string) bool {
    return IsProfane(s, 1)
}

Where

level	description	example
1	low filter	fuck, bitch
2	moderate filter	boobs, ass
3	strict filter	adult, blood, period

* Level 2 inherits from level 1 and level 3 inherits from level 2

Note that the strict filter would have a lot of false positive, but its goal would be to prevent as much bad things from getting by as possible, even at the cost of those false positives.

The point here is that by default, the level 1 filter should be used, but if you don't trust where the strings are coming from at all, then you can scale the filter's strictness based on the level of trust.

False positive: "button"

Describe the bug

A substring of the word is inappropriately censored.

What do you see?

When censoring the string "button", the profanity detector censors a substring which results in "****on".

What do you expect to see?

When censoring the string "button", the profanity detector should identify the word as a false positive and leave it unchanged.

List the steps that must be taken to reproduce this issue

1. Initialise a Go codebase
2. Create a Go file with a main function
3. Install the "github.com/TwiN/go-away" library
4. Import the "fmt" package from the Go standard library
5. Import the "github.com/TwiN/go-away" library
6. Call the goaway.Censor function on the string "button" and use the fmt.Println function to output it to the terminal
7. Run the code

Version

v1.6.12

Additional information

Reference to a similar issue: #66
PR for a fix: #68

bug: incorrect position replaced

Describe the bug

version: 1.6.9
test:

import (
	"testing"

	goaway "github.com/TwiN/go-away"
	"github.com/stretchr/testify/require"
)

func TestDefaultDriver_AR_4768(t *testing.T) {
	detector := goaway.NewProfanityDetector().WithCustomDictionary(
		[]string{"anal", "babi"}, // profanities
		[]string{"babibu"},       // falsePositives
		[]string{"babibube"},     // falseNegatives
	)

	message := "message with old anal new babi falsePositive babibu falseNegative babibube"
	expected := "message with old **** new **** falsePositive babibu falseNegative ********"

	isProfane := detector.IsProfane(message)
	require.True(t, isProfane)

	censored := detector.Censor(message)
	require.Equal(t, expected, censored)

	/*
		Original :message with old anal new babi falsePositive babibu falseNegative babibube
		Expected :message with old **** new **** falsePositive babibu falseNegative ********
		Actual   :message with old anal ne* ***i fa****ositive babibu falseNegative ********
	*/
}

What do you see?

message with old **** new **** falsePositive babibu falseNegative ********

What do you expect to see?

message with old **** new **** falsePositive babibu falseNegative ********

List the steps that must be taken to reproduce this issue

use the test from the description

Version

v1.6.9

Additional information

No response

Fix (False Positive): Tassilo

Describe the feature request

Add
"tassi", // tassilo, tassie
to the list of false positives

Why do you personally want this feature to be implemented?

Some of our userbase have the name Tassilo.
Also found Tassie is a noun

How long have you been using this project?

Over a year!

Additional information

No rush, thanks as always.
Are you allowing contributors?
I couldn't open a PR

Support for multiple languages

Describe the feature request

Would be cool if we have support for multiple languages! I'm down to write it for Portuguese and Spanish.

Why do you personally want this feature to be implemented?

Have multi-language in one library

How long have you been using this project?

No response

Additional information

No response

Allow using different profanities

We discovered your library because we are looking to resolve the same problem that the library does, so was great to see a very full library that cover a lot of case when you want to discover bad words, but we saw that the library didn't support use a different dictionary of profanities so that is a problem is you maintain a multi-language site.

So could be nice that the tool support a to load of different profanities, falseNegatives and falsePositives, maybe loading a different ones on the construction of the ProfanityDetector with a new functional function.

I took the freedom that also implement a proposal for that, so I will send you a PR

Thanks!

Add censor method

Replaces profanity, in a string, with asterisks.

Censor("go fuck yourself"); // go **** yourself

Index out of range error when censoring cyrillic language

Describe the bug

When i try to censor a word in cyrillic it throw a panic error

What do you see?

panic: runtime error: index out of range [8] with length 8

goroutine 62 [running]:

github.com/TwiN/go-away.(*ProfanityDetector).Censor(0xc00039bd28, {0xc0001a0c10, 0x10})
        /home/user/.gvm/pkgsets/go1.18rc1/global/pkg/mod/github.com/!twi!n/go-away@v1.6.0/goaway.go:179 +0x6d1

What do you expect to see?

A correct censorship of the word

List the steps that must be taken to reproduce this issue

filter := goaway.NewProfanityDetector().WithCustomDictionary(profanities, fp, fn)

cleanedWord := f.Censor("крывавы")

Version

1.6.0

Additional information

No response

False positive: "therapeutic"

Describe the bug

A substring of the word is censored.

A workaround is to create a new profanity detector with a custom dictionary, in which the word is appended to the set of false positives.

What do you see?

When censoring the string "therapeutic", the profanity detector censors a substring which results in "the****utic".
The string "therapist" is listed as a false positive, but not "therapeutic".

What do you expect to see?

When censoring the string "therapeutic", the profanity detector should identify the word as a false positive and leave it unchanged.

List the steps that must be taken to reproduce this issue

1. Initialise a Go codebase
2. Create a Go file with a main function
3. Install the "github.com/TwiN/go-away" library
4. Import the "fmt" package from the Go standard library
5. Import the "github.com/TwiN/go-away" library
6. Call the goaway.Censor function on the string "therapeutic" and use the fmt.Println function to output it to the terminal
7. Run the code

Version

No response

Additional information

No response

False Positives: Kassie, Kassidy, Kassandra

Describe the bug

Names in subject are incorrectly identified as profanity.

What do you see?

"Kassie, "Kassandra", "Kassidy", etc. are identified as profanity.

What do you expect to see?

These names should not be identified as profanity.

List the steps that must be taken to reproduce this issue

1. Check "Kassie" for profanity.
2. See that IsProfane returns true.

Version

v1.6.4

Additional information

I propose that "kass" is added to the falsepostives.go file. I'm willing to make this addition and update tests if given permission.

bug: Custom Dictionaries Uppercased fails to match

Describe the bug

We use a custom dictionary for other things I was hoping to use the same dictionary here. The issue is this dictionary is all upper case. I'm not in a position to change the dictionary used everywhere else so I'm needing this library to Lower the dictionary word as it runs

What do you see?

ran with the following input list words := []string{"FUCK", "ass", "POOP", "penis", "bitch"}
and this custom Dictionary WithCustomDictionary([]string{"FUCK", "ASS", "POOP", "PENIS", "BITCH"}, DefaultFalsePositives, DefaultFalseNegatives)
im getting the following output
Expected true, got false from word FUCK
Expected true, got false from word ass
Expected true, got false from word POOP
Expected true, got false from word penis
Expected true, got false from word bitch

What do you expect to see?

I expeted it to pass the test in all cases

List the steps that must be taken to reproduce this issue

1. use custom dictionary containing uppercase words
2. Run the test

Version

latest

Additional information

I have a functional fork that i would like to bring in where i am ToLower the dictionary words as they come through the Extract profanity function

False positives: "can use", "via ssh"

Of course, I could add these to the false positives list, but maybe there's a better, more general way to tackle these.

Add butterfly to false positives

Describe the feature request

Add butterfly to false positives

Why do you personally want this feature to be implemented?

Usernames with butterfly are blocked 🦋

How long have you been using this project?

No response

Additional information

Thanks so much!

Add a list of false positives to reduce unjustified filtering

When a word contains a profanity, like assassins, the word could be the check inside a dictionary.

bug: panic, when using utf-8

Describe the bug

use the test:

func TestDefaultDriver_UTF8(t *testing.T) {
	detector := goaway.NewProfanityDetector().WithCustomDictionary(
		[]string{"anal", "あほ"}, // profanities
		[]string{"あほほ"},        // falsePositives
		[]string{"あほほし"},       // falseNegatives
	)

	message := "いい加減にしろ あほほし あほほ あほ anal ほ"
	expected := "いい加減にしろ **** あほほ ** **** ほ"

	isProfane := detector.IsProfane(message)
	require.True(t, isProfane)

	censored := detector.Censor(message)
	require.Equal(t, expected, censored)
}

What do you see?

panic at:
github.com/TwiN/go-away.(*ProfanityDetector).Censor(0xc000082f08, {0xf8f7c7, 0x3c})
.../github.com/TwiN/go-away/goaway.go:193 +0x7a5

What do you expect to see?

UTF-8 filtered as well as ASCII

List the steps that must be taken to reproduce this issue

use the test from the description

Version

v1.6.9

Additional information

No response

bug: runtime error: index out of range [-1] when using custom profanity list

Describe the bug

Panic error on profanity free message:
"Ядром сообщества является команда"

Error log:

2022/12/10 16:06:23 http: panic serving 127.0.0.1:55219: runtime error: index out of range [-1]
goroutine 167 [running]:
net/http.(*conn).serve.func1()
        /usr/local/go/src/net/http/server.go:1850 +0xb0
panic({0x1043c9860, 0x14004732060})
        /usr/local/go/src/runtime/panic.go:890 +0x258
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.func1()
        /Users/oneart/go/pkg/mod/go.opentelemetry.io/otel/[email protected]/trace/span.go:393 +0x2c
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0x140045ee780, {0x0, 0x0, 0x7700684000817cc0?})
        /Users/oneart/go/pkg/mod/go.opentelemetry.io/otel/[email protected]/trace/span.go:432 +0x6e8
panic({0x1043c9860, 0x14004732060})
        /usr/local/go/src/runtime/panic.go:884 +0x204
github.com/TwiN/go-away.(*ProfanityDetector).Censor(0x140005834a0, {0x140008174c0, 0x3f})
        /Users/oneart/go/pkg/mod/github.com/!twi!n/[email protected]/goaway.go:193 +0x564
github.com/oneart-dev/****/services.publicProfileService.Update({{}, {0x140005834a0}, {0x14000023ab0}, {0x140001248d0}, {0x1044e23f0, 0x1400124ad80}, {0x1044dab90, 0x14001361e60}, {0x1044ef0e0, 0x1400016a100}, ...}, ...)

What do you see?

Panic

What do you expect to see?

It should not touch the string as no profanity in this words

List the steps that must be taken to reproduce this issue

I'm sorry to post this words here :)

profanitiesAll := []string{
		"блять", "блядь", "бляди", "блядина", "блядище", "блядский", "блядство", "блядун", "блядуны", "блядунья", "блядюга", "блядюги", "блядюха", "блядюшка", "блядюшки", "блядюшник", "блядюшница", "блядюшницы", "блядюшничать", "блядюшничество", "блядюшный", "архипиздрит", "басран", "бздение", "бздеть", "бздех", "бзднуть", "бздун", "бздунья", "бздюха", "бикса", "блежник", "блудилище", "бляд", "блябу", "блябуду", "блядун", "блядунья", "блядь", "блядюга", "взьебка", "волосянка", "взьебывать", "вз'ебывать", "выблядок", "выблядыш", "выебать", "выеть", "выпердеть", "высраться", "выссаться", "говенка", "говенный", "говешка", "говназия", "говнецо", "говно", "говноед", "говночист", "говнюк", "говнюха", "говнядина", "говняк", "говняный", "говнять", "гондон", "дермо", "долбоеб", "дрисня", "дрист", "дристать", "дристануть", "дристун", "дристуха", "дрочена", "дрочила", "дрочилка", "дрочить", "дрочка", "ебало", "ебальник", "ебануть", "ебаный", "ебарь", "ебатория", "ебать", "ебаться", "ебец", "ебливый", "ебля", "ебнуть", "ебнуться", "ебня", "ебун", "елда", "елдак", "елдачить", "заговнять", "задристать", "задрока", "заеба", "заебанец", "заебать", "заебаться", "заебываться", "заеть", "залупа", "залупаться", "залупить", "залупиться", "замудохаться", "засерун", "засеря", "засерать", "засирать", "засранец", "засрун", "захуячить", "злоебучий", "изговнять", "изговняться", "кляпыжиться", "курва", "курвенок", "курвин", "курвяжник", "курвяжница", "курвяжный", "манда", "мандавошка", "мандей", "мандеть", "мандища", "мандюк", "минет", "минетчик", "минетчица", "мокрохвостка", "мокрощелка", "мудак", "муде", "мудеть", "мудила", "мудистый", "мудня", "мудоеб", "мудозвон", "муйня", "набздеть", "наговнять", "надристать", "надрочить", "наебать", "наебнуться", "наебывать", "нассать", "нахезать", "нахуйник", "насцать", "обдристаться", "обдристаться", "обосранец", "обосрать", "обосцать", "обосцаться", "обсирать", "опизде", "отпиздячить", "отпороть", "отъеть", "охуевательский", "охуевать", "охуевающий", "охуеть", "охуительный", "охуячивать", "охуячить", "педрик", "пердеж", "пердение", "пердеть", "пердильник", "перднуть", "пердун", "пердунец", "пердунина", "пердунья", "пердуха", "пердь", "передок", "пернуть", "пидор", "пизда", "пиздануть", "пизденка", "пиздеть", "пиздить", "пиздища", "пиздобратия", "пиздоватый", "пиздорванец", "пиздорванка", "пиздострадатель", "пиздун", "пиздюга", "пиздюк", "пиздячить", "писять", "питишка", "плеха", "подговнять", "подъебнуться", "поебать", "поеть", "попысать", "посрать", "поставить", "поцоватый", "презерватив", "проблядь", "проебать", "промандеть", "промудеть", "пропиздеть", "пропиздячить", "пысать", "разъеба", "разъебай", "распиздай", "распиздеться", "распиздяй", "распроеть", "растыка", "сговнять", "секель", "серун", "серька", "сика", "сикать", "сикель", "сирать", "сирывать", "скурвиться", "скуреха", "скурея", "скуряга", "скуряжничать", "спиздить", "срака", "сраный", "сранье", "срать", "срун", "ссака", "ссаки", "ссать", "старпер", "струк", "суходрочка", "сцавинье", "сцака", "сцаки", "сцание", "сцать", "сциха", "сцуль", "сцыха", "сыкун", "титечка", "титечный", "титка", "титочка", "титька", "трипер", "триппер", "уеть", "усраться", "усцаться", "фик", "фуй", "хезать", "хер", "херня", "херовина", "херовый", "хитрожопый", "хлюха", "хуевина", "хуевый", "хуек", "хуепромышленник", "хуерик", "хуесос", "хуище", "хуй", "хуйня", "хуйрик", "хуякать", "хуякнуть", "целка", "шлюха",
	}

	profanitiesAll = append(profanitiesAll, goaway.DefaultProfanities...)
	falsePositives := goaway.DefaultFalsePositives
	falseNegatives := goaway.DefaultFalsePositives

	profanityDetector := goaway.NewProfanityDetector().WithCustomDictionary(profanitiesAll, falsePositives, falseNegatives)

profanityDetector.Censor("Ядром сообщества является команда")

Version

1.6.8

Additional information

If I remove my list of profanities all seems to work fine.

go test -race is issuing WARNING: DATA RACE in IsProfane()

Describe the bug

I'm using TwiN/go-away in a user model for checking profanity in the username.

Tests for functionality pass (thank you for this library)!

When I turn on go test -race, I receive warnings in IsProfane():

==================
WARNING: DATA RACE
Read at 0x000101ce9d08 by goroutine 31:
  github.com/TwiN/go-away.IsProfane()
      /Users/damondanieli/go/pkg/mod/github.com/!twi!n/[email protected]/goaway.go:294 +0x34
  github.com/tikr-media/tikr-services/models.validateUsername()
      /Users/damondanieli/go/src/github.com/tikr-media/tikr-services/models/user.go:220 +0x190
  github.com/tikr-media/tikr-services/models.(*userModel).CreateUser()
      /Users/damondanieli/go/src/github.com/tikr-media/tikr-services/models/user.go:96 +0xa8
  github.com/tikr-media/tikr-services/models_test.TestUsername.func1()
      /Users/damondanieli/go/src/github.com/tikr-media/tikr-services/models/user_test.go:70 +0x3f0
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.17.6/libexec/src/testing/testing.go:1259 +0x198

Previous write at 0x000101ce9d08 by goroutine 32:
  github.com/TwiN/go-away.IsProfane()
      /Users/damondanieli/go/pkg/mod/github.com/!twi!n/[email protected]/goaway.go:295 +0x2d8
  github.com/tikr-media/tikr-services/models.validateUsername()
      /Users/damondanieli/go/src/github.com/tikr-media/tikr-services/models/user.go:220 +0x190
  github.com/tikr-media/tikr-services/models.(*userModel).CreateUser()
      /Users/damondanieli/go/src/github.com/tikr-media/tikr-services/models/user.go:96 +0xa8
  github.com/tikr-media/tikr-services/models_test.TestUserWithSQLite.func1.1()
      /Users/damondanieli/go/src/github.com/tikr-media/tikr-services/models/user_test.go:23 +0x90
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.17.6/libexec/src/testing/testing.go:1259 +0x198

Goroutine 31 (running) created at:
  testing.(*T).Run()
      /opt/homebrew/Cellar/go/1.17.6/libexec/src/testing/testing.go:1306 +0x5b8
  github.com/tikr-media/tikr-services/models_test.TestUsername()
      /Users/damondanieli/go/src/github.com/tikr-media/tikr-services/models/user_test.go:60 +0x230
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.17.6/libexec/src/testing/testing.go:1259 +0x198

Goroutine 32 (running) created at:
  testing.(*T).Run()
      /opt/homebrew/Cellar/go/1.17.6/libexec/src/testing/testing.go:1306 +0x5b8
  github.com/tikr-media/tikr-services/models_test.TestUserWithSQLite.func1()
      /Users/damondanieli/go/src/github.com/tikr-media/tikr-services/models/user_test.go:22 +0x1e0
  testing.tRunner()
      /opt/homebrew/Cellar/go/1.17.6/libexec/src/testing/testing.go:1259 +0x198
==================

What do you see?

No response

What do you expect to see?

No response

List the steps that must be taken to reproduce this issue

1. Add go-away into a test.
2. Call IsProfane() from multiple tests.
3. Set tests to run in parallel (t.Parallel())
4. Run with race conditions go test -race ./your_tests/.

Version

v1.6.0

Additional information

This is the only library I'm using that issues these warnings.

Support toggling leet speak

First of all, thanks so much for making this plugin!

However, I am wracking my brain as to how the following 3 "words" are being flagged as profane.

fmt.Println(goaway.IsProfane("BwMAF4GiogA")) //true
fmt.Println(goaway.IsProfane("BwMBF4GiogA")) //true
fmt.Println(goaway.IsProfane("BwMCF4GiogA")) //true

Duplicate false negatives and profanities break censoring

Describe the bug

When a string has two false negatives and two profanities, the string is not censored properly.

What do you see?

println(goaway.Censor("document fuck document fuck"))

outputs

document **** ****ment fuck

For verbosity, note that by default, document is a false positive (meaning it should not be censored) and fuck is a profanity.

What do you expect to see?

The output should be:

document **** document ****

List the steps that must be taken to reproduce this issue

1. Uncomment the test case that can be found at

   go-away/goaway_test.go

   Lines 148 to 152 in 3bf4c7a

   	//{
   	// input: "document fuck document fuck", // FIXME: This is not censored properly
   	// expectedOutput: "document **** document ****",
   	// expectedOutputWithoutSpaceSanitization: "document **** document ****",
   	//},

2. Run the tests
3. Note how the new test case is failing

Version

v1.6.3

Additional information

I found this out while investigating #30