rstudio / graphframes Goto Github PK
View Code? Open in Web Editor NEWR Interface for GraphFrames
Home Page: https://spark.rstudio.com/graphframes/
License: Apache License 2.0
R Interface for GraphFrames
Home Page: https://spark.rstudio.com/graphframes/
License: Apache License 2.0
graphframes 0.6 was released few days back, which comes with support to spark 2.2+. it may make sense to update the dependency to graphframes v0.6 in place of v0.5.
I have checked everywhere but it seems that graphframes in R dont have aggregateMessages function.
Does anyone know any R ways for aggregateMessages?
Hi . I try to start with graphframe . My code is
library(graphframes)
library(sparklyr)
library(dplyr)
# connect to spark using sparklyr
sc <- spark_connect(master = "local", version = "2.3.0")
But I get the following error
Error in force(code) :
Failed while connecting to sparklyr to port (8880) for sessionid (7002): Gateway in port (8880) did not respond.
When I start sparklyr without import graphframes spark inits ok.
I've already seen this issue in sparklyr but I don't know how to solve. Any idea ?
Hi,
I am trying to follow the instructions under https://spark.rstudio.com/graphframes/ for running graphframes
with spark version 2.1.0
.
However, I am facing a similar issue as has been described before in #7.
That means after:
sparklyr::spark_install(version = "2.1.0")
I can connect to spark in a fresh R session via:
library(sparklyr)
sc <- spark_connect(master = "local", version = "2.1.0")
However, when also loading graphframes
, I would run into the following error:
> library(sparklyr)
> library(graphframes)
> sc <- spark_connect(master = "local", version = "2.1.0", config = conf)
Ivy Default Cache set to: /Users/ludwig/.ivy2/cache
The jars for the packages stored in: /Users/ludwig/.ivy2/jars
:: loading settings :: url = jar:file:/Users/ludwig/spark/spark-2.1.0-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
graphframes#graphframes added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
:: resolution report :: resolve 1226ms :: artifacts dl 0ms
:: modules in use:
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 0 | 0 | 0 || 0 | 0 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
module not found: graphframes#graphframes;0.5.0-spark2.1-s_2.11
==== local-m2-cache: tried
file:/Users/ludwig/.m2/repository/graphframes/graphframes/0.5.0-spark2.1-s_2.11/graphframes-0.5.0-spark2.1-s_2.11.pom
-- artifact graphframes#graphframes;0.5.0-spark2.1-s_2.11!graphframes.jar:
file:/Users/ludwig/.m2/repository/graphframes/graphframes/0.5.0-spark2.1-s_2.11/graphframes-0.5.0-spark2.1-s_2.11.jar
==== local-ivy-cache: tried
/Users/ludwig/.ivy2/local/graphframes/graphframes/0.5.0-spark2.1-s_2.11/ivys/ivy.xml
-- artifact graphframes#graphframes;0.5.0-spark2.1-s_2.11!graphframes.jar:
/Users/ludwig/.ivy2/local/graphframes/graphframes/0.5.0-spark2.1-s_2.11/jars/graphframes.jar
==== central: tried
https://repo1.maven.org/maven2/graphframes/graphframes/0.5.0-spark2.1-s_2.11/graphframes-0.5.0-spark2.1-s_2.11.pom
-- artifact graphframes#graphframes;0.5.0-spark2.1-s_2.11!graphframes.jar:
https://repo1.maven.org/maven2/graphframes/graphframes/0.5.0-spark2.1-s_2.11/graphframes-0.5.0-spark2.1-s_2.11.jar
==== spark-packages: tried
http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.5.0-spark2.1-s_2.11/graphframes-0.5.0-spark2.1-s_2.11.pom
-- artifact graphframes#graphframes;0.5.0-spark2.1-s_2.11!graphframes.jar:
http://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.5.0-spark2.1-s_2.11/graphframes-0.5.0-spark2.1-s_2.11.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: UNRESOLVED DEPENDENCIES ::
::::::::::::::::::::::::::::::::::::::::::::::
:: graphframes#graphframes;0.5.0-spark2.1-s_2.11: not found
::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: graphframes#graphframes;0.5.0-spark2.1-s_2.11: not found]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1078)
at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:296)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:160)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Error in spark_connect_gateway(gatewayAddress, gatewayPort, sessionId, :
Gateway in localhost:8880 did not respond.
I have tried using a more recent version of spark (2.4.3) as well as putting the apparently missing graphframes jars directly into the jars
directory without success.
Any advice on how to resolve this would be greatly appreciated. Thanks!
> sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 10.16
Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] dplyr_1.0.7 graphframes_0.1.2 sparklyr_1.7.0
loaded via a namespace (and not attached):
[1] pillar_1.6.2 compiler_4.1.0 BiocManager_1.30.16
[4] dbplyr_2.1.1 prettyunits_1.1.1 remotes_2.4.0
[7] r2d3_0.2.5 base64enc_0.1-3 tools_4.1.0
[10] pkgbuild_1.2.0 digest_0.6.27 jsonlite_1.7.2
[13] lifecycle_1.0.0 tibble_3.1.3 pkgconfig_2.0.3
[16] rlang_0.4.11 cli_3.0.1 DBI_1.1.1
[19] rstudioapi_0.13 curl_4.3.2 yaml_2.2.1
[22] parallel_4.1.0 withr_2.4.2 httr_1.4.2
[25] generics_0.1.0 vctrs_0.3.8 htmlwidgets_1.5.3
[28] askpass_1.1 rappdirs_0.3.3 rprojroot_2.0.2
[31] tidyselect_1.1.1 glue_1.4.2 forge_0.2.0
[34] R6_2.5.0 processx_3.5.2 fansi_0.5.0
[37] callr_3.7.0 purrr_0.3.4 tidyr_1.1.3
[40] magrittr_2.0.1 ps_1.6.0 ellipsis_0.3.2
[43] htmltools_0.5.1.1 assertthat_0.2.1 config_0.3.1
[46] utf8_1.2.2 openssl_1.4.4 crayon_1.4.1
Looks like SVD++ is implemented in Scala's graphframe but not in the R interface.
The link to graphframes in the README.md for this repo is broken. Should probably point to
https://graphframes.github.io/graphframes/docs/_site/index.html
Can you provide some guidance on how to use this with spark 3.0/graphframes 0.8.1? I do not
see doc on how to add graphframes classes to classpath so that sparklyr can find them. Undoubtedly
I should have more spark configuration knowledge, but if it is easy to describe this it would be
appreciated.
related to thomasp85/tidygraph#86
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.