Spark@Stackoverflow

Apache Spark questions on StackOverflow. Run by @huitseeker

[Spark@Stackoverflow @SparkAtSO]

Skewed dataset join in Spark?: I ma joining two big datasets using Spark RDD, one dataset is very much skewed... https://t.co/Ir58nsQmus

[Spark@Stackoverflow @SparkAtSO]

Structured streaming in spark for streaming machine learning algorithms: I am beginner for spark streaming. S... https://t.co/Oqxn8vg7OQ

[Spark@Stackoverflow @SparkAtSO]

How to set spark.local.dir property from spark shell?: I'm trying to set spark.local.dir from spark-shell usi... https://t.co/h90MGKCeQG

[Spark@Stackoverflow @SparkAtSO]

how to encrypt & Decrypt database credentials in spark sql: Here i have mentioned my user name and password e... https://t.co/h4K3YKMUiA

[Spark@Stackoverflow @SparkAtSO]

How to read and write data to Google Bigtable in pyspark application ?: I am using Cloudproc spark cluster wh... https://t.co/j5BIUlwfAH

[Spark@Stackoverflow @SparkAtSO]

How to specify sql dialect when creating spark dataframe from JDBC?: I'm having an issue reading data via cus... https://t.co/DvKm75BLVK

[Spark@Stackoverflow @SparkAtSO]

Talend and Apache Spark?: I am confused as to where Talend and Apache spark fit in the big data ecosystem as ... https://t.co/7d5qmKnrU2

[Spark@Stackoverflow @SparkAtSO]

Spark Pivot - generates a matrix: I need to pivot a two-column dataframe to one row (long to wide) so that I ... https://t.co/IKmbQr1UOF

[Spark@Stackoverflow @SparkAtSO]

Using python scipy from spark scala: Is it possible to use python "scipy" linear algebra library from spark/s... https://t.co/EJ0N6YREZc

[Spark@Stackoverflow @SparkAtSO]

aggregating existing data by ids from mongodb in spark: I'm wondering if Mongo-spark is able to handle a scen... https://t.co/G1Sd2D7vNU

[Spark@Stackoverflow @SparkAtSO]

java.lang.UnsupportedOperationException: Schema for type MyClass is not supported: I am using Spark 1.5.0 and... https://t.co/nnAYo9YqQd

[Spark@Stackoverflow @SparkAtSO]

Registering Hive Custom UDF with Spark (Spark SQL) 2.0.0: I am working on a spark 2.0.0 piece where my requir... https://t.co/Pg0BwQGHhd

[Spark@Stackoverflow @SparkAtSO]

Creating Pyspark DataFrame column that coalesces two other Columns, why am I getting error of 'unicode' objec... https://t.co/pZEgLaV7qc

[Spark@Stackoverflow @SparkAtSO]

How to get rid of the row wrapper object in pyspark dataframes without using RDD api?: I'm issuing the follow... https://t.co/zSOq8KtKGQ

[Spark@Stackoverflow @SparkAtSO]

When does spark-env.sh runs?: I am assuming that spark-env.sh runs when I start the spark cluster with

sbin... https://t.co/OvRRuDsm17

[Spark@Stackoverflow @SparkAtSO]

Spark Cassandra String not written in Hadoop file: I am Java Developer and learning Spark while working on a ... https://t.co/7UWQ2iIdJe

[Spark@Stackoverflow @SparkAtSO]

Partitioning in Spark: I created an RDD by parallelizing the following array:

var arr: Array[(Int,Char)] = A... https://t.co/92BZJ5Qof2

[Spark@Stackoverflow @SparkAtSO]

Running EMR Spark with multiple S3 accounts: I have an EMR Spark Job that needs to read data from S3 on one a... https://t.co/bvlSWjWzh7

[Spark@Stackoverflow @SparkAtSO]

Why does a Spark Application launch with only a single executor on DC/OS?: I have Spark installed, but when I... https://t.co/OZQxPDjvve

[Spark@Stackoverflow @SparkAtSO]

Hive/SparkSQL - Auto typecast date to timestamp in expression: When I run Hive queries in Spark SQL, LHS (tim... https://t.co/0qgXqrj9Ib