TOP

解決Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/ FSDataInputStream
2018-11-22 08:37:02 】 瀏覽:1214
Tags:

我們將 Hadoop 的 classhpath 信息添加到 CLASSPATH 變量中优乐棋牌app下载,在 ~/.bashrc 中增加如下幾行:</span>

export HADOOP_HOME=/home/hadoop/app/hadoop

export JAVA_HOME=/home/hadoop/app/java/jdk

exportSCALA_HOME=/home/hadoop/app/scala
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:$PATH
export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath):$CLASSPATH

spark-env.sh中添加環境變量:

export JAVA_HOME=/home/hadoop/app/jdk
export SCALA_HOME=/home/hadoop/app/scala
export SPARK_MASTER_IP=zhangge
export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/home/hadoop/app/hadoop/etc/hadoop
export SPARK_DIST_CLASSPATH=$(hadoop classpath)//非常重要的變量。可參考下面的鏈接解釋

https://spark.apache.org/docs/latest/hadoop-provided.html#using-sparks-hadoop-free-build

Using Spark's "Hadoop Free" Build
Spark uses Hadoop client libraries for HDFS and YARN. Starting in version Spark 1.4, the project packages “Hadoop free” builds that lets you more easily connect a single Spark binary to any Hadoop version. To use these builds, you need to modify SPARK_DIST_CLASSPATH to include Hadoop’s package jars. The most convenient place to do this is by adding an entry in conf/spark-env.sh.
This page describes how to connect Spark to Hadoop for different types of distributions.
Apache Hadoop
For Apache distributions, you can use Hadoop’s ‘classpath’ command. For instance:

Apache Hadoop

For Apache distributions, you can use Hadoop’s ‘classpath’ command. For instance:

### in conf/spark-env.sh ###
# If 'hadoop' binary is on your PATH
export SPARK_DIST_CLASSPATH=$(hadoop classpath)
# With explicit path to 'hadoop' binary
export SPARK_DIST_CLASSPATH=$(/path/to/hadoop/bin/hadoop classpath)
# Passing a Hadoop configuration directory
export SPARK_DIST_CLASSPATH=$(hadoop --config /path/to/configs classpath)

請關注公眾號獲取更多資料



】【打印繁體】【】【】 【】【】【】 【關閉】 【返回頂部
上一篇hadoop學習心得之安裝01   .. 下一篇hadoop   建立單點集群