問題描述

我花了幾天時間嘗試讓 Spark 與我的 Jupyter Notebook 和 Anaconda 一起工作.這是我的 .bash_profile 的樣子:

I've spent a few days now trying to make Spark work with my Jupyter Notebook and Anaconda. Here's what my .bash_profile looks like:

PATH="/my/path/to/anaconda3/bin:$PATH"

export JAVA_HOME="/my/path/to/jdk"
export PYTHON_PATH="/my/path/to/anaconda3/bin/python"
export PYSPARK_PYTHON="/my/path/to/anaconda3/bin/python"

export PATH=$PATH:/my/path/to/spark-2.1.0-bin-hadoop2.7/bin
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook" pyspark
export SPARK_HOME=/my/path/to/spark-2.1.0-bin-hadoop2.7
alias pyspark="pyspark --conf spark.local.dir=/home/puifais --num-executors 30 --driver-memory 128g --executor-memory 6g --packages com.databricks:spark-csv_2.11:1.5.0"

當(dāng)我輸入 /my/path/to/spark-2.1.0-bin-hadoop2.7/bin/spark-shell 時，我可以在命令行 shell 中正常啟動 Spark.并且輸出 sc 不為空.它似乎工作正常.

When I type /my/path/to/spark-2.1.0-bin-hadoop2.7/bin/spark-shell, I can launch Spark just fine in my command line shell. And the output sc is not empty. It seems to work fine.

當(dāng)我輸入 pyspark 時，它會啟動我的 Jupyter Notebook.當(dāng)我創(chuàng)建一個新的 Python3 筆記本時，會出現(xiàn)這個錯誤:

When I type pyspark, it launches my Jupyter Notebook fine. When I create a new Python3 notebook, this error appears:

[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /my/path/to/spark-2.1.0-bin-hadoop2.7/python/pyspark/shell.py: 

而我的 Jupyter Notebook 中的 sc 是空的.

And sc in my Jupyter Notebook is empty.

誰能幫忙解決這個問題?

Can anyone help solve this situation?

只是想澄清一下:錯誤末尾的冒號后面沒有任何內(nèi)容.我還嘗試使用此 post 創(chuàng)建我自己的啟動文件，我在這里引用，所以你不必去看那里:

Just want to clarify: There is nothing after the colon at the end of the error. I also tried to create my own start-up file using this post and I quote here so you don't have to go look there:

我創(chuàng)建了一個簡短的初始化腳本 init_spark.py，如下所示:

I created a short initialization script init_spark.py as follows:

from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("yarn-client")
sc = SparkContext(conf = conf)

并將其放在 ~/.ipython/profile_default/startup/目錄中

and placed it in the ~/.ipython/profile_default/startup/ directory

當(dāng)我這樣做時，錯誤就變成了:

When I did this, the error then became:

[IPKernelApp] WARNING | Unknown error in handling PYTHONSTARTUP file /my/path/to/spark-2.1.0-bin-hadoop2.7/python/pyspark/shell.py:
[IPKernelApp] WARNING | Unknown error in handling startup files:

推薦答案

Conda 可以幫助正確管理很多依賴...

Conda can help correctly manage a lot of dependencies...

安裝火花.假設(shè) spark 安裝在/opt/spark 中，請將其包含在您的 ~/.bashrc 中:

Install spark. Assuming spark is installed in /opt/spark, include this in your ~/.bashrc:

export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH

創(chuàng)建一個 conda 環(huán)境，其中包含除 spark 之外的所有所需依賴項(xiàng):

Create a conda environment with all needed dependencies apart from spark:

conda create -n findspark-jupyter-openjdk8-py3 -c conda-forge python=3.5 jupyter=1.0 notebook=5.0 openjdk=8.0.144 findspark=1.1.0

激活環(huán)境

$ source activate findspark-jupyter-openjdk8-py3

啟動 Jupyter Notebook 服務(wù)器:

Launch a Jupyter Notebook server:

$ jupyter notebook

在您的瀏覽器中，創(chuàng)建一個新的 Python3 筆記本

In your browser, create a new Python3 notebook

嘗試使用以下腳本計(jì)算 PI(借自 this)

Try calculating PI with the following script (borrowed from this)

import findspark
findspark.init()
import pyspark
import random
sc = pyspark.SparkContext(appName="Pi")
num_samples = 100000000
def inside(p):     
  x, y = random.random(), random.random()
  return x*x + y*y < 1
count = sc.parallelize(range(0, num_samples)).filter(inside).count()
pi = 4 * count / num_samples
print(pi)
sc.stop()

這篇關(guān)于配置 Spark 以使用 Jupyter Notebook 和 Anaconda的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網(wǎng)！