GithubHelp home page GithubHelp logo

bruin-data / bruin Goto Github PK

View Code? Open in Web Editor NEW
60.0 60.0 2.0 27.66 MB

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

Home Page: https://bruin-data.github.io/bruin/

License: Apache License 2.0

Makefile 0.02% Go 9.66% Python 90.32%
analytics bigquery data-analysis data-modeling data-pipelines data-transformation python snowflake sql

bruin's People

Contributors

albertobruin avatar batikankarakan avatar karakanb avatar sabrikaragonen avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

ishan-marikar

bruin's Issues

Error While Running Pipeline Locally: Concurrent Map Writes 

(nimbostratus_venv) oem@machine:~/Documents/Github/reponame$ bruin run pipelines/daily/
Analyzed the pipeline 'daily' with 3 assets.

Pipeline: daily (.)
  No issues found

✓ Successfully validated 3 tasks across 1 pipeline, all good.

Starting the pipeline execution...

[2024-01-18 16:46:45] Starting: schema_external.weather_historical
[2024-01-18 16:46:45] Starting: schema_external.market_price
[2024-01-18 16:46:45] Starting: schema_external.dayahead_prediction
fatal error: concurrent map writes

goroutine 13 [running]:
github.com/bruin-data/bruin/pkg/python.(*LocalOperator).RunTask(0xc000a2e320, {0x22838f8, 0xc000baddd0}, 0xc000a7cf20, 0xc000b5e700)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/python/operator.go:132 +0x2d3
github.com/bruin-data/bruin/pkg/python.(*LocalOperator).Run(0x1b444a0?, {0x22838f8, 0xc000baddd0}, {0x2296788?, 0xc000ba38f0})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/python/operator.go:92 +0x85
github.com/bruin-data/bruin/pkg/executor.Sequential.RunSingleTask({0x22836c8?}, {0x22838f8, 0xc000baddd0}, {0x2296788?, 0xc000ba38f0?})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/sequential.go:38 +0x1ab
github.com/bruin-data/bruin/pkg/executor.worker.run({{0xc000bafe08, 0x8}, 0xc000122be0, 0xc000122a28, 0xc0000b36e0, 0xc000bafdf0}, 0x0?, 0x0?)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:101 +0x348
created by github.com/bruin-data/bruin/pkg/executor.Concurrent.Start in goroutine 1
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:73 +0x75

goroutine 1 [select]:
github.com/bruin-data/bruin/pkg/scheduler.(*Scheduler).Run(0xc00001d740, {0x22836c8, 0x318ff00})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/scheduler/scheduler.go:395 +0x185
github.com/bruin-data/bruin/cmd.Run.func1(0xc00001c100)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/cmd/run.go:250 +0x17f8
github.com/urfave/cli/v2.(*Command).Run(0xc0006209a0, 0xc00001c100, {0xc000021fa0, 0x2, 0x2})
	/home/oem/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:277 +0x9d8
github.com/urfave/cli/v2.(*Command).Run(0xc000621080, 0xc00001c000, {0xc000118810, 0x3, 0x3})
	/home/oem/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:270 +0xc25
github.com/urfave/cli/v2.(*App).RunContext(0xc0001b0960, {0x22836c8?, 0x318ff00}, {0xc000118810, 0x3, 0x3})
	/home/oem/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:335 +0x5db
github.com/urfave/cli/v2.(*App).Run(...)
	/home/oem/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:309
main.main()
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/main.go:37 +0x6f7

goroutine 7 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000546280)
	/home/oem/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/oem/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 8 [IO wait]:
internal/poll.runtime_pollWait(0x7f07647c4e58, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000547580?, 0xc000128ec0?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).ReadMsg(0xc000547580, {0xc000128ec0, 0x10, 0x10}, {0xc000816020, 0x1000, 0x1000}, 0x3179480?)
	/usr/local/go/src/internal/poll/fd_unix.go:301 +0x38a
net.(*netFD).readMsg(0xc000547580, {0xc000128ec0?, 0xc000823ce0?, 0x41b4d3?}, {0xc000816020?, 0x0?, 0x100000001?}, 0x0?)
	/usr/local/go/src/net/fd_posix.go:78 +0x31
net.(*UnixConn).readMsg(0xc0005925d8, {0xc000128ec0?, 0x426b7c?, 0x3192340?}, {0xc000816020?, 0x7f07647eb318?, 0xc0000408a0?})
	/usr/local/go/src/net/unixsock_posix.go:115 +0x45
net.(*UnixConn).ReadMsgUnix(0xc0005925d8, {0xc000128ec0?, 0x41f2b6?, 0x1010003192340?}, {0xc000816020?, 0x412305?, 0xc00076e7c8?})
	/usr/local/go/src/net/unixsock.go:143 +0x36
github.com/godbus/dbus.(*oobReader).Read(0xc000816000, {0xc000128ec0?, 0xc000823e30?, 0x412685?})
	/home/oem/go/pkg/mod/github.com/godbus/[email protected]/transport_unix.go:21 +0x3c
io.ReadAtLeast({0x226ff00, 0xc000816000}, {0xc000128ec0, 0x10, 0x10}, 0x10)
	/usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
	/usr/local/go/src/io/io.go:354
github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc00069c078)
	/home/oem/go/pkg/mod/github.com/godbus/[email protected]/transport_unix.go:91 +0xfe
github.com/godbus/dbus.(*Conn).inWorker(0xc000812090)
	/home/oem/go/pkg/mod/github.com/godbus/[email protected]/conn.go:294 +0x35
created by github.com/godbus/dbus.(*Conn).Auth in goroutine 1
	/home/oem/go/pkg/mod/github.com/godbus/[email protected]/auth.go:118 +0x9a5

goroutine 11 [IO wait]:
internal/poll.runtime_pollWait(0x7f07647c4b70, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc000b60180?, 0xc000bb4000?, 0x1)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000b60180, {0xc000bb4000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
os.(*File).read(...)
	/usr/local/go/src/os/file_posix.go:29
os.(*File).Read(0xc000122088, {0xc000bb4000?, 0xaf?, 0x8000?})
	/usr/local/go/src/os/file.go:118 +0x52
io.copyBuffer({0x2268e60, 0xc00073a048}, {0x2268c20, 0xc000122088}, {0x0, 0x0, 0x0})
	/usr/local/go/src/io/io.go:430 +0x1a6
io.Copy(...)
	/usr/local/go/src/io/io.go:389
github.com/bruin-data/bruin/cmd.logOutput.func1()
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/cmd/run.go:419 +0x38
created by github.com/bruin-data/bruin/cmd.logOutput in goroutine 1
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/cmd/run.go:417 +0x445

goroutine 12 [runnable]:
bytes.growSlice.func1()
	/usr/local/go/src/bytes/buffer.go:230 +0x4b
bytes.growSlice({0xc000c98200, 0x3c, 0x4a063b?}, 0x94d?)
	/usr/local/go/src/bytes/buffer.go:251 +0x11e
bytes.(*Buffer).grow(0xc000590c80, 0x9a5)
	/usr/local/go/src/bytes/buffer.go:151 +0x13d
bytes.(*Buffer).Write(0xc000590c80, {0xc000bce000, 0x9a5, 0x73ab27?})
	/usr/local/go/src/bytes/buffer.go:179 +0x59
encoding/json.stringEncoder(0xc000590c80, {0x1a60bc0?, 0xc000704120?, 0x0?}, {0xfe?, 0x7e?})
	/usr/local/go/src/encoding/json/encode.go:591 +0x3dd
encoding/json.mapEncoder.encode({0x0?}, 0xc000590c80, {0x1afa760?, 0xc000040900?, 0x1afa760?}, {0xd8?, 0xf6?})
	/usr/local/go/src/encoding/json/encode.go:759 +0x566
encoding/json.(*encodeState).reflectValue(0xc000590c80?, {0x1afa760?, 0xc000040900?, 0x412685?}, {0x38?, 0x0?})
	/usr/local/go/src/encoding/json/encode.go:323 +0x73
encoding/json.(*encodeState).marshal(0x1c53660?, {0x1afa760?, 0xc000040900?}, {0xb8?, 0x25?})
	/usr/local/go/src/encoding/json/encode.go:295 +0xb9
encoding/json.Marshal({0x1afa760, 0xc000040900})
	/usr/local/go/src/encoding/json/encode.go:162 +0xd0
github.com/bruin-data/bruin/pkg/config.GoogleCloudPlatformConnection.MarshalJSON({{0xc000bae4b0, 0xb}, {0xc000bc2a80, 0x94e}, {0xc000a54300, 0x25}, {0xc000046a50, 0x14}, 0x0})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/config/manager.go:43 +0x16e
encoding/json.marshalerEncoder(0xc000590740, {0x1b8eb40?, 0xc000a2e0a0?, 0x22b0250?}, {0x10?, 0x81?})
	/usr/local/go/src/encoding/json/encode.go:442 +0xd2
encoding/json.typeEncoder.func1(0x1b8eb40?, {0x1b8eb40?, 0xc000a2e0a0?, 0xc000590740?}, {0x40?, 0x0?})
	/usr/local/go/src/encoding/json/encode.go:360 +0x7b
encoding/json.(*encodeState).reflectValue(0xc000590740?, {0x1b8eb40?, 0xc000a2e0a0?, 0x412685?}, {0x38?, 0x0?})
	/usr/local/go/src/encoding/json/encode.go:323 +0x73
encoding/json.(*encodeState).marshal(0xc000bbeb98?, {0x1b8eb40?, 0xc000a2e0a0?}, {0x0?, 0x0?})
	/usr/local/go/src/encoding/json/encode.go:295 +0xb9
encoding/json.Marshal({0x1b8eb40, 0xc000a2e0a0})
	/usr/local/go/src/encoding/json/encode.go:162 +0xd0
github.com/bruin-data/bruin/pkg/config.(*Environment).GetSecretByKey(0x64?, {0xc000a0b212?, 0xc000c96150?})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/config/manager.go:123 +0x65
github.com/bruin-data/bruin/pkg/config.(*Config).GetSecretByKey(0x0?, {0xc000a0b212?, 0x27?})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/config/manager.go:138 +0x1c
github.com/bruin-data/bruin/pkg/python.(*LocalOperator).RunTask(0xc000a2e320, {0x22838f8, 0xc000040570}, 0xc000a7cf20, 0xc000b5e540)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/python/operator.go:123 +0x295
github.com/bruin-data/bruin/pkg/python.(*LocalOperator).Run(0x1b444a0?, {0x22838f8, 0xc000040570}, {0x2296788?, 0xc000ba3880})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/python/operator.go:92 +0x85
github.com/bruin-data/bruin/pkg/executor.Sequential.RunSingleTask({0x22836c8?}, {0x22838f8, 0xc000040570}, {0x2296788?, 0xc000ba3880?})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/sequential.go:38 +0x1ab
github.com/bruin-data/bruin/pkg/executor.worker.run({{0xc000bafdf8, 0x8}, 0xc000122be0, 0xc000122a28, 0xc0000b36c0, 0xc000bafdf0}, 0x1805565?, 0xc000812090?)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:101 +0x348
created by github.com/bruin-data/bruin/pkg/executor.Concurrent.Start in goroutine 1
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:73 +0x75

goroutine 14 [chan receive]:
github.com/bruin-data/bruin/pkg/executor.worker.run({{0xc000bafe18, 0x8}, 0xc000122be0, 0xc000122a28, 0xc0000b3700, 0xc000bafdf0}, 0x0?, 0x0?)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:86 +0xa5
created by github.com/bruin-data/bruin/pkg/executor.Concurrent.Start in goroutine 1
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:73 +0x75

goroutine 15 [runnable]:
github.com/bruin-data/bruin/pkg/executor.(*workerWriter).Write(0xc000c82150, {0xc0002144c0, 0x3f, 0x225d7b0?})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:130 +0xe8
github.com/bruin-data/bruin/pkg/python.log({0x22838f8, 0xc000c82180}, {0x1df55b2, 0x3e})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/python/local.go:42 +0x10d
github.com/bruin-data/bruin/pkg/python.(*localPythonRunner).Run(0xc000badbf0, {0x22838f8, 0xc000c82180}, 0xc0000301c0)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/python/local.go:56 +0x1ab
github.com/bruin-data/bruin/pkg/python.(*LocalOperator).RunTask(0xc000a2e320, {0x22838f8, 0xc000c82180}, 0xc000a7cf20, 0xc000b5e380)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/python/operator.go:135 +0x567
github.com/bruin-data/bruin/pkg/python.(*LocalOperator).Run(0x1b444a0?, {0x22838f8, 0xc000c82180}, {0x2296788?, 0xc000ba3810})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/python/operator.go:92 +0x85
github.com/bruin-data/bruin/pkg/executor.Sequential.RunSingleTask({0x22836c8?}, {0x22838f8, 0xc000c82180}, {0x2296788?, 0xc000ba3810?})
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/sequential.go:38 +0x1ab
github.com/bruin-data/bruin/pkg/executor.worker.run({{0xc000bafe28, 0x8}, 0xc000122be0, 0xc000122a28, 0xc0000b3720, 0xc000bafdf0}, 0x0?, 0x0?)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:101 +0x348
created by github.com/bruin-data/bruin/pkg/executor.Concurrent.Start in goroutine 1
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:73 +0x75

goroutine 16 [chan receive]:
github.com/bruin-data/bruin/pkg/executor.worker.run({{0xc000bafe38, 0x8}, 0xc000122be0, 0xc000122a28, 0xc0000b3740, 0xc000bafdf0}, 0x0?, 0x0?)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:86 +0xa5
created by github.com/bruin-data/bruin/pkg/executor.Concurrent.Start in goroutine 1
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:73 +0x75

goroutine 66 [chan receive]:
github.com/bruin-data/bruin/pkg/executor.worker.run({{0xc000bafe48, 0x8}, 0xc000122be0, 0xc000122a28, 0xc0000b3760, 0xc000bafdf0}, 0x0?, 0x0?)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:86 +0xa5
created by github.com/bruin-data/bruin/pkg/executor.Concurrent.Start in goroutine 1
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:73 +0x75

goroutine 67 [chan receive]:
github.com/bruin-data/bruin/pkg/executor.worker.run({{0xc000bafe58, 0x8}, 0xc000122be0, 0xc000122a28, 0xc0000b3780, 0xc000bafdf0}, 0x0?, 0x0?)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:86 +0xa5
created by github.com/bruin-data/bruin/pkg/executor.Concurrent.Start in goroutine 1
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:73 +0x75

goroutine 68 [chan receive]:
github.com/bruin-data/bruin/pkg/executor.worker.run({{0xc000bafe68, 0x8}, 0xc000122be0, 0xc000122a28, 0xc0000b37a0, 0xc000bafdf0}, 0x0?, 0x0?)
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:86 +0xa5
created by github.com/bruin-data/bruin/pkg/executor.Concurrent.Start in goroutine 1
	/home/oem/go/pkg/mod/github.com/bruin-data/[email protected]/pkg/executor/concurrent.go:73 +0x75

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.