Spark Trouble shooting

Increasing the resultset size:

• spark-submit: --conf spark.driver.maxResultSize=4g
• conf.set("spark.driver.maxResultSize", "4g")
• spark-defaults.conf: spark.driver.maxResultSize 4g

Useful Amazon EC2 commands

SSH to EC2 instance:
sudo ssh -i <path/cluster.pem> user@public DNS

SCP file from desktop to EC2 instance: