Skip to content

Conversation

jegonzal
Copy link
Contributor

We had originally disabled reference tracking by default. However,this creates serious issues in the spark-shell. For example the following code will fail:

class A(a: String) extends Serializable
val x = sc.parallelize(Array.fill(10)(new A("hello")))
x.collect

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@AmplabJenkins
Copy link

Merged build finished. All automated tests passed.

@AmplabJenkins
Copy link

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14359/

@rxin
Copy link
Contributor

rxin commented Apr 24, 2014

Is the problem A has a reference to the outer level repl object? Maybe a better solution here is to turn reference tracking off when we serialize closures in the shell mode ... @mateiz thoughts?

@mateiz
Copy link
Contributor

mateiz commented Apr 25, 2014

We could do that too, but how would we detect that we're in the shell here? Maybe one way is to set the track references SparkConf field to false and don't override it here if it is set.

@mateiz
Copy link
Contributor

mateiz commented May 12, 2014

@jegonzal @rxin is this still needed for GraphX to work in the shell in 1.0 or do you guys have a workaround?

@mateiz
Copy link
Contributor

mateiz commented May 12, 2014

Actually it looks like this will be subsumed by #724. You should close this pull request, as GitHub won't automatically close it.

@jegonzal jegonzal closed this May 12, 2014
pwendell pushed a commit to pwendell/spark that referenced this pull request May 12, 2014
Replace commons-math with jblas in SVDPlusPlus
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants