GithubHelp home page GithubHelp logo

lintool / bespin Goto Github PK

View Code? Open in Web Editor NEW
81.0 81.0 96.0 209 KB

Reference implementations of data-intensive algorithms in MapReduce and Spark

Home Page: http://bespin.io/

License: Other

Java 78.65% Scala 21.35%

bespin's People

Contributors

adrw avatar lintool avatar luochenhuan avatar youngbink avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

blueskywalker aaroc357 m4haley j367liu y89zhu evayyho alexandra03 owenchen bmuscede colourfulblank ashwin1593 michwu salihkardan moore-ryan youngbink nupurkumar tabithahaha n0740 dlng wazibaba caohy1988 asteriq zhumzhu yantaigao giserh rosequ wanxingwang coderneil afaurot leahdineen wjwmichael hellowsummer shadowridgedev tailaijin yanghan11 vitorianavi aimjwizards h35gao fmxu3720 skeftical yangjiezhou yi-wang djwierds esther09 bagelseed luke202001 hans-zhang28 adrw lidiyam vincent-weng benjaminll aayushshah15 paopao74cn jyxt alivivi thava annanw4201 ganding martinhu1994 sunnydjinn fliu777 danielkou harkamaljot 0n0dera tracezhoudrrr aixeuy maggie4v4 cosine37 leme-fg harry-l lsyfelicia reynoldsm88 jiawenhou rudranib istiaq107 jaski-waterloo ccyung758 l9sharma big-data-teaching jinyanghuang nedhuang aabedii harry688tan96 wwwheat carlosvv gameloser segfault2017 jeffreyfei 4d11 kathyrinelu pvfrota geekhusky stariqmi dmusican linnall

bespin's Issues

Error of WordCount exmaple

When I did the initial steps to run wordcount example by executing

hadoop jar target/bespin-1.0.1-SNAPSHOT-fatjar.jar io.bespin.scala.mapreduce.wordcount.WordCount --input data/Shakespeare.txt --output wc-smr-combiner

I got this error message:

17/04/03 14:10:38 INFO wordcount.WordCount$: Input: data/Shakespeare.txt

17/04/03 14:10:38 INFO wordcount.WordCount$: Output: wc-smr-combiner

17/04/03 14:10:38 INFO wordcount.WordCount$: Number of reducers: 1

17/04/03 14:10:38 INFO wordcount.WordCount$: Use in-mapper combining: false

17/04/03 14:10:38 INFO wordcount.WordCount$: Use in-mapper histogram: false

Exception in thread "main" java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V

at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:131)

at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:139)

at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:206)

at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:185)

at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:237)

at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:522)

at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:505)

at org.apache.hadoop.mapreduce.JobContext.<init>(JobContext.java:80)

at org.apache.hadoop.mapreduce.Job.<init>(Job.java:100)

at org.apache.hadoop.mapreduce.Job.getInstance(Job.java:71)

at io.bespin.scala.mapreduce.wordcount.WordCount$.run(WordCount.scala:152)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

at io.bespin.scala.mapreduce.wordcount.WordCount$.main(WordCount.scala:184)

at io.bespin.scala.mapreduce.wordcount.WordCount.main(WordCount.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

This link said it might be the confliction of packages

http://stackoverflow.com/questions/37116688/nosuchmethoderror-org-slf4j-spi-locationawarelogger-log

I am wondering if I am the first one got this problem and how can I solve it.

Recurrent misspelling of "adjacency" as "adjaceny"

Examples:

Build command throws 501 HTTPS Required error

The current build command mvn clean package results in a build failure due to 501 errors

Detailed error:

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33.970 s
[INFO] Finished at: 2020-09-13T15:40:40-04:00
[INFO] Final Memory: 38M/1514M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project bespin: Could not resolve dependencies for project io.bespin:bespin:jar:1.0.5-SNAPSHOT: Failed to collect dependencies at org.apache.spark:spark-core_2.11:jar:2.3.1 -> net.java.dev.jets3t:jets3t:jar:0.9.4 -> commons-codec:commons-codec:jar:1.15-SNAPSHOT: Failed to read artifact descriptor for commons-codec:commons-codec:jar:1.15-SNAPSHOT: Could not transfer artifact commons-codec:commons-codec:pom:1.15-SNAPSHOT from/to maven (http://repo.maven.apache.org/maven2/): Failed to transfer file: http://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.15-SNAPSHOT/commons-codec-1.15-SNAPSHOT.pom. Return code is: 501 , ReasonPhrase:HTTPS Required. -> [Help 1]

This is because of the following change and can be fixed by using https in the repository tag of pom.xml :

  <repositories>
    <repository>
      <id>maven</id>
      <url>http://repo.maven.apache.org/maven2/</url> -> <url>https://repo.maven.apache.org/maven2/</url>
    </repository>
  </repositories>

source: this was reported and resolved on the CS451 Piazza, I just copied it over here

Loss of log probability precision in PageRank implementation

The reference implementation of PageRank throws away accuracy when calculating missing mass (RunPageRankBasic.java:456) by bringing the log-probability back into linear space to compute the missing mass.

Obviously, this introduces error proportional to how small the missing mass is, as well as how many iterations are ran. Specifically, it will slightly affect the solution values for an assignment of a certain systems course taught using this library - 0.00001 on public local test cases, and potentially more on blind cloud test cases. (It does not affect the values in the README example.)

The patch is here (too lazy to pull-request): https://pastebin.com/A9gV6jRi

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.