GithubHelp home page GithubHelp logo

portbase's People

Contributors

dependabot[bot] avatar dhaavi avatar ppacher avatar qbit avatar raphty avatar raskaro avatar vlabo avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar

Forkers

fossabot backup-fork qbit classicvalues mrsinghchem llschuster rohankumardubey mrabbehassan axionize

portbase's Issues

Implement MaintainRecordStates in badger and fstree storages

portbase/database/storage/badger/badger.go

Lines 210 to 214 in a0ec633

// MaintainRecordStates maintains records states in the database.
func (b *Badger) MaintainRecordStates(ctx context.Context, purgeDeletedBefore time.Time) error {
// TODO: implement MaintainRecordStates
return nil
}

portbase/database/storage/fstree/fstree.go

Lines 268 to 272 in a0ec633

// MaintainRecordStates maintains records states in the database.
func (fst *FSTree) MaintainRecordStates(ctx context.Context, purgeDeletedBefore time.Time) error {
// TODO: implement MaintainRecordStates
return nil
}

Modules test sometimes time out on Windows

Sometimes the module tests fail - see https://travis-ci.org/github/safing/portbase/jobs/706840823.

Failing test:

[ FAIL ] go test -cover -short github.com/safing/portbase/modules
taking too long

goroutine 22 [running]:
runtime/pprof.writeGoroutineStacks(0x5eb3a0, 0xc0000bc010, 0x30, 0xd0)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/runtime/pprof/pprof.go:665 +0xa4
runtime/pprof.writeGoroutine(0x5eb3a0, 0xc0000bc010, 0x2, 0x0, 0xc0000be6d0)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/runtime/pprof/pprof.go:654 +0x4b
runtime/pprof.(*Profile).WriteTo(0x6f6d80, 0x5eb3a0, 0xc0000bc010, 0x2, 0x1, 0x10)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/runtime/pprof/pprof.go:329 +0x3e1
github.com/safing/portbase/modules.init.3.func1()
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks_test.go:20 +0xf5
created by github.com/safing/portbase/modules.init.3
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks_test.go:17 +0x6c

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00015a7e0, 0x5b7ce1, 0x13, 0x5c2538, 0x48a201)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:1043 +0x385
testing.runTests.func1(0xc00015a000)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:1284 +0x7f
testing.tRunner(0xc00015a000, 0xc0000b9dd8)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:991 +0xe3
testing.runTests(0xc00008b140, 0x6fae00, 0x8, 0x8, 0x0)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:1282 +0x2ae
testing.(*M).Run(0xc0000ee280, 0x0)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:1199 +0x166
main.main()
	_testmain.go:130 +0x1c8

goroutine 19 [select]:
github.com/safing/portbase/modules.microTaskScheduler()
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/microtasks.go:198 +0x2d5
created by github.com/safing/portbase/modules.init.2
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/microtasks_test.go:18 +0x3c

goroutine 20 [semacquire]:
sync.runtime_Semacquire(0x72dba8)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/runtime/sema.go:56 +0x49
sync.(*WaitGroup).Wait(0x72dba0)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/sync/waitgroup.go:130 +0x6b
github.com/safing/portbase/modules.taskQueueHandler()
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks.go:449 +0x8f
created by github.com/safing/portbase/modules.init.3
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks_test.go:14 +0x3c

goroutine 21 [select]:
github.com/safing/portbase/modules.taskScheduleHandler()
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks.go:489 +0x10d
created by github.com/safing/portbase/modules.init.3
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks_test.go:15 +0x54

goroutine 23 [chan send]:
github.com/safing/portbase/modules.init.3.func2()
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks_test.go:27 +0x3e
created by github.com/safing/portbase/modules.init.3
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks_test.go:25 +0x84

goroutine 26 [chan receive]:
testing.(*T).Parallel(0xc00015a360)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:867 +0x1dc
github.com/safing/portbase/modules.TestModules(0xc00015a360)
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/modules_test.go:49 +0x36
testing.tRunner(0xc00015a360, 0x5c2530)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:991 +0xe3
created by testing.(*T).Run
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:1042 +0x35e

goroutine 32 [semacquire]:
sync.runtime_Semacquire(0xc000090d38)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/runtime/sema.go:56 +0x49
sync.(*WaitGroup).Wait(0xc000090d30)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/sync/waitgroup.go:130 +0x6b
github.com/safing/portbase/modules.TestQueueSuccession(0xc00015a7e0)
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks_test.go:250 +0xee
testing.tRunner(0xc00015a7e0, 0x5c2538)
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:991 +0xe3
created by testing.(*T).Run
	C:/Users/travis/.gimme/versions/go1.14.4.windows.amd64/src/testing/testing.go:1042 +0x35e

goroutine 12 [select]:
github.com/safing/portbase/modules.(*Task).runWithLocking.func1(0xc000048000)
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks.go:313 +0xe5
created by github.com/safing/portbase/modules.(*Task).runWithLocking
	C:/Users/travis/gopath/src/github.com/safing/portbase/modules/tasks.go:312 +0x31e

FAIL	github.com/safing/portbase/modules	30.067s
FAIL

Use JWT for API Authentication Tokens

I guess it would be easier to use a JWT here: generate a random key at the beginning and use that key to sign/encrypt JWT issued to clients. This way we can remove the global state validTokens which will make unit testing easier and we also don't need to care about token cleanup. The JWT RFC already defines an "expiresAt" field so we don't need any global state to verify if a token is valid or not.

If we start issuing long-lived API tokens JWT would also be a better choice because we can simply store the signing key instead of tracking each and every token.

Originally posted by @ppacher in #55

Inconsistent database accessor behavior

When trying to store a struct with json:"someThing" field tags in the database the behavior of the accessor function used during query processing is inconsistent as the actual JSON blob (and thus JSON field accessor) has a different key than the struct based accessor (which uses the Field name as it is).

Consider the following record type:

type Foo struct {
    record.Base
    Name string `json:"name"`
}

A query that searches for name == "foo" must look like this to work as expected:

where := query.Or(
    query.Where("name", query.SameAs, "foo"), // required for the JSON value accessor
    query.Where("Name", query.SameAs, "foo"), // required for the Struct field accessor
)

This adds considerably more complexity when writing database queries.

Proposed solutions

  1. Drop the different accessor functions and replace them by a common one.
    AFAIK the different accessor functions have been added for performance reasons as it's not required to decode the JSON of each database record. We can either check if this really poses a performance issue (or if it has been assumed to be one) and if, check if implementations like https://github.com/segmentio/encoding would reduce the overhead enough to perform JSON decoding on every record.
    We might also consider to keep the JSON accessor and marshal any already-parsed record back to JSON. Which one is better likely depends on cache size.

  2. Update the struct-based accessor to support JSON tags.
    While being relatively simple to achieve, we add an additional dependency to JSON and will likely increase the burden of switching to some different encoding in the future.

Add database/query support for time.Time

We currently resort to using int64 instead of time.Time for database records when we plan on using these fields in a query. If we would support time.Time in the query package, we could utilize this more.

Return error directly if PutMany is not supported.

Currently, PutMany returns a function that returns an error if the Batcher interface is not supported by the storage. It would be a lot cleaner to have PutMany also return an error directly if such an early error occurs.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.