Recent questions tagged scala

0 votes

1.3k views

1 answer

scala - how to filter out a null value from spark dataframe

I created a dataframe in spark with the following schema: root |-- user_id: long (nullable = false) |-- event_id: ... how can I fix it? Thanks See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - How to set heap size for sbt?

I am using SBT 0.12.0. I have read other answers on stack overflow and followed them, however none of them ... when doing both testing and run? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - How to write unit tests in Spark 2.0+?

I've been trying to find a reasonable way to test SparkSession with the JUnit testing framework. While there ... sensible error will be thrown. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - How to create SBT project with IntelliJ Idea?

I just got started with Scala/LiftWeb/Sbt developing, and I'd like to import a Sbt project in IntelliJ Idea. ... Can anyone help me with this? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.5k views

1 answer

scala - @karate How to pass parameter to a feature file in gatling simulation class?

Let's consider a scenario, we have to run the performance test for "create an account api" which takes ... functional test to performance test. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Apache Spark how to append new column from list/array to Spark dataframe

I am using Apache Spark 2.0 Dataframe/Dataset API I want to add a new column to my dataframe from List ... in reality contains more columns. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - How to get ClassTag form TypeTag, or both at same time?

I have some code like this: class ReflectiveJsonFormat[T:TypeTag] extends JsonFormat[T] { def write(x: T) : ... you otherwise fix this code? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Why doesn't Array's == function return true for Array(1,2) == Array(1,2)?

In Programming in Scala the authors write that Scala's == function compares value equality instead of reference ... function work on arrays? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - Why can't I return a concrete subtype of A if a generic subtype of A is declared as return parameter?

abstract class IntTree object Empty extends IntTree case class NonEmpty(elem: Int, left: IntTree, right: IntTree) ... am I missing here? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - call of distinct and map together throws NPE in spark library

I am unsure if this is a bug, so if you do something like this // d:spark.RDD[String] d.distinct().map(x => d ... fine. I am using spark 0.6.1. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.5k views

1 answer

scala - How do I make a function involving futures tail recursive?

In my Scala app, I have a function that calls a function which returns a result of type Future[T]. I ... FactorialCalc.factorial(4), 5.seconds) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Explanation of fold method of spark RDD

I am running Spark-1.4.0 pre-built for Hadoop-2.4 (in local mode) to calculate the sum of squares ... explain what exactly is happening here? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.1k views

1 answer

scala - How to get path to the uploaded file

I am running an spark cluster on google cloud and I upload a configuration file with each job. What is ... .properties --jars my.jar See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.5k views

1 answer

scala - What are the rules to govern underscore to define anonymous function?

I am using _ as placeholder for creating anonymous function, and the problem is I cannot predict how Scala is going ... this? What's the rule ? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

2.4k views

1 answer

scala - Result type of an implicit conversion must be more specific than AnyRef

Let def h(a: AnyRef*) = a.mkString(",") h: (a: AnyRef*)String and so h("1","2") res: String = 1,2 ... 1 and 2.11.1. To ask on a workaround. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - No Json serializer as JsObject found for type play.api.libs.json.JsObject

I have the following code that works in a console app when referencing "org.reactivemongo" %% "play2-reactivemongo" % ... What am I doing wrong? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Is there a way to control which implicit conversion will be the default used?

Suppose I have this: class String2(val x:String) { def *(times:Int) : String = { val builder = new StringBuilder( ... ), but I think it will do. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - What is going wrong with `unionAll` of Spark `DataFrame`?

Using Spark 1.5.0 and given the following code, I expect unionAll to union DataFrames based on their column name ... Sounds like a serious bug!? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Run custom task automatically before/after standard task

I often want to do some customization before one of the standard tasks are run. I realize I can make new ... a standard tasks are invoked ?. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - How to convert unix timestamp to date in Spark

I have a data frame with a column of unix timestamp(eg.1435655706000), and I want to convert it to data ... " is malformed at "6000" See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.7k views

1 answer

scala - Cartesian product of two lists

Given a map where a digit is associated to several characters scala> val conversion = Map("0" -> List(" ... same result for any sequence length? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.6k views

1 answer

scala - How to use Column.isin with list?

val items = List("a", "b", "c") sqlContext.sql("select c1 from table") .filter($"c1".isin(items)) ... ".isin(items)) .collect .foreach(println) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - Better way to convert a string field into timestamp in Spark

I have a CSV in which a field is datetime in a specific format. I cannot import it directly in my ... the schema for the Dataframe again. See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - Syntax sugar: _* for treating Seq as method parameters

I just noticed this construct somewhere on web: val list = List(someCollection: _*) What does _* mean? Is ... advantage of this syntax sugar? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.0k views

1 answer

scala - Stackoverflow due to long RDD Lineage

I have thousands of small files in HDFS. Need to process a slightly smaller subset of files (which is ... .apply(TraversableLike.scala:244) See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.2k views

1 answer

scala - java.io.IOException: Could not locate executable nullinwinutils.exe in the Hadoop binaries. spark Eclipse on windows 7

I'm not able to run a simple spark job in Scala IDE (Maven spark project) installed on Windows 7 Spark ... .FrameDemo.main(FrameDemo.scala)<br> See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.3k views

1 answer

scala - how do I get sbt to use a local maven proxy repository (Nexus)?

I've got an sbt (Scala) project that currently pulls artifacts from the web. We'd like to move towards a ... -directory is not viable.) Thanks! See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

0 votes

1.4k views

1 answer

scala - Shuffle a list of integers with Java 8 Streams API

I tried to translate the following line of Scala to Java 8 using the Streams API: // Scala util.Random.shuffle ... list with Java 8 Streams API? See Question&Answers more detail:os...

asked Oct 17, 2021 in Technique[技术] by 深蓝 (71.8m points)

Categories

Just Browsing Browsing

Most popular tags