Comments (4)
The variable NUM_MAPS
is configured for workload running, not preparing.
from hibench.
thanks for reply...then how to configure to execute my scenario?
from hibench.
Sorry, my last comment seems to be wrong after I saw the code.
from hibench.
@brahmareddybattula Can u paste the printed message like:
Submit MapReduce Job: /home/lv/intel/cluster/hadoop/hadoop-2.5.0-cdh5.3.2/bin/hadoop --config /home/lv/intel/cluster/hadoop/hadoop-2.5.0-cdh5.3.2/etc/hadoop jar /home/lv/intel/cluster/hadoop/hadoop-2.5.0-cdh5.3.2/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.5.0-cdh5.3.2.jar randomtextwriter -D mapreduce.randomtextwriter.totalbytes=32000000000 -D mapreduce.job.maps=12 -D mapreduce.job.reduces=6 -D mapreduce.output.fileoutputformat.compress=false hdfs://lv-dev:54310/HiBench/Wordcount/Input
or check your report/wordcount/prepare/conf/wordcount.conf
for NUM_MAPS
, and report/wordcount/prepare/conf/sparkbench/sparkbench.conf
for hibench.default.map.parallelism
?
Number of mappers should follow the configurations as you defined and report/wordcount/prepare/conf/wordcount.conf
will tell you what and where the value of NUM_MAPS has been defined. Take mine as an example:
# Source: /home/lv/intel/HiBench/conf/99-user_defined_properties.conf
HADOOP_HOME=/home/lv/intel/cluster/hadoop/hadoop-2.5.0-cdh5.3.2
HDFS_MASTER=hdfs://lv-dev:54310
NUM_MAPS=12
NUM_REDS=6
SPARK_HOME=/home/lv/intel/cluster/spark/spark-1.3.0-bin-hadoop2.4
SPARK_MASTER=yarn-client
YARN_EXECUTOR_CORES=4
YARN_NUM_EXECUTORS=4
from hibench.
Related Issues (20)
- can you add me pls
- I am facing the below mentioned issue while trying to execute the pagerank algorithm in hadoop from hibench. Please give a solution. HOT 1
- upgrade kafka version
- Hi
- HiveError:HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient HOT 1
- Does Hibench Suite work with spark version 3.2.1 ? HOT 1
- Hi
- Error when run sql test
- Does Hibench Suite support Hadoop version 3.x ? HOT 1
- i don't no what i do -_- HOT 2
- Could tasks submitted to Spark by Hibench be viewed in Spark's History Server?
- Issues with run-sparkbench on Google Cloud Platform (GCP)
- spark config value modification does not apply.
- for single node hadoop configuration
- Hi
- The
- ERROR [26/26] RUN cd /root/HiBench && mvn clean package -Dspark=1.6 -Dscala=2.10
- Attempt to run HiBench on JDK 17 or higher versions HOT 1
- Hi
- Runstreaming job failed with error in bench.log like "org.apache.spark.SparkException: Error getting partition metadata for 'identity'. Does the topic exist?"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hibench.