Comments (12)
package main
import (
"flag"
"path/filepath"
"strings"
"os"
"github.com/chrislusf/gleam/flow"
"github.com/chrislusf/gleam/gio"
"github.com/chrislusf/gleam/plugins/kafka"
)
var (
brokers = flag.String("brokers", "beta-hbase02:9092,beta-hbase03:9092,beta-hbase04:9092", "a list of comma separated broker:port")
topic = flag.String("topic", "test", "the topic name")
group = flag.String("group", filepath.Base(os.Args[0]), "the consumer group name")
timeout = flag.Int("timeout", 30, "the number of seconds for timeout connections")
)
func main() {
gio.Init()
flag.Parse()
brokerList := strings.Split(*brokers, ",")
k := kafka.New(brokerList, *topic, *group)
k.TimeoutSeconds = *timeout
f := flow.New("kafka " + *topic).Read(k).Printlnf("%x")
f.Run()
}
from gleam.
k is like this:
from gleam.
k = {*github.com/chrislusf/gleam/plugins/kafka.KafkaSource | 0xc0000dc7d0}
Brokers = {[]string} len:3, cap:3
0 = {string} "beta-hbase02:9092"
1 = {string} "beta-hbase03:9092"
2 = {string} "beta-hbase04:9092"
Group = {string} "___go_build_gocrawler_gleamtest_standalone_kafkareader"
Topic = {string} "test"
TimeoutSeconds = {int} 16
prefix = {string} "test"
from gleam.
try to add some actual work between read and println.
from gleam.
added a Map between read and println, still no print, debug with break point set in the 1st line of capitalize, no stop after new msgs sent into the topic.
from gleam.
Capitalize = gio.RegisterMapper(capitalize)
)
func main() {
gio.Init()
flag.Parse()
brokerList := strings.Split(*brokers, ",")
k := kafka.New(brokerList, *topic, *group)
k.TimeoutSeconds = *timeout
f := flow.New("kafka " + *topic).Read(k).Map("capitalize", Capitalize).Printlnf("%x")
f.Run()
}
func capitalize(row []interface{}) error {
line := gio.ToString(row[0])
gio.Emit(strings.ToUpper(line))
return nil
}
from gleam.
i tried connect to a not existing topic, the topic appear after gleam app is started, but nothing active detected in the gleam app, here is the summary after app is stopped:
from gleam.
step:testgleam.list0
output : d0
shard:0 time:306.505µs completed 285
step:testgleam.Read.Map1
input : d0
shard:0 time:306.505µs completed 285
output : d1
shard:0 time:1m16.258485573s processed 0
step:capitalize.Map2
input : d1
shard:0 time:1m16.258495764s processed 0
output : d2
shard:0 time:1m16.258422547s processed 0
from gleam.
BTW, default partition num in the remote kafka cluster is 12
from gleam.
or if there is a QQ group for gleam discussion, pls. share the id
from gleam.
connect to some real kafka topic. There are some buffers. If not much traffic, the data are buffered.
from gleam.
great! after connect to a real topic in production, it is working now. how to control the buffer, i would like each new msg is dealt by the Map ASAP without buffering.
from gleam.
Related Issues (20)
- how to set parallesim when in distributed mode HOT 2
- Performance benchmarks
- any way similiar to flink/storm keyedBy to have data specific for 1 user always go into same single partition/thread? HOT 2
- wordcount executed in testing framework with EOF errors HOT 3
- Is it possible to register mappers dynamically? HOT 4
- Does gleam have 32 bit system support? HOT 1
- Performance Optimization is required HOT 2
- Possible copy-paste error message HOT 1
- Possible risk of temporary file collision HOT 1
- Optimisation: S3/GS files shouldn't need to touch disk
- Support for reading multiple files from S3
- backward incompatible change in github.com/xitongsys/parquet-go HOT 1
- .
- [Feature] Supporting SeaweedFS with Gleam
- Are there any consumption cases of RocketMQ
- Question: Query To AST parser HOT 1
- Is this project active? HOT 2
- K8s executor failed to start: "no such file or directory" HOT 1
- Running Gleam Tasks external to a gleam cluster HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gleam.