开始C++在Hadoop上的开发,之前一直Java.发现也要废点周折。网上的一些资料也比较少,可能台简单了吧。这里记录下。或许也有像我一样的同学,需要Java,C++兼顾.
按照网上的示例开始运行word count.按照步骤通过命令ant -Dcompile.c++=yes examples编译pipes examles时发现如下报错:
[exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:428: undefined reference to `HMAC_Final' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:429: undefined reference to `HMAC_CTX_cleanup' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:435: undefined reference to `BIO_f_base64' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:435: undefined reference to `BIO_new' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:436: undefined reference to `BIO_s_mem' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:436: undefined reference to `BIO_new' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:437: undefined reference to `BIO_push' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:438: undefined reference to `BIO_write' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:439: undefined reference to `BIO_ctrl' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:440: undefined reference to `BIO_ctrl' [exec] /home/guoyun/workspace/hadoop-0.20.2-cdh3u0/src/c++/pipes/impl/HadoopPipes.cc:445: undefined reference to `BIO_free_all' [exec] collect2: ld returned 1 exit status [exec] make: *** [wordcount-simple] Error 1
搜了一圈没搜到什么,大多是一般的C++语法问题,编译的时候加个参数-lcrypto。但怎么应用到hadoop中呢?兜了下,还是找到了方案。只要在$HADOOP_HOME/src/examples/pipes/Makefile.in中修改
LDADD = -L$(HADOOP_UTILS_PREFIX)/lib -L$(HADOOP_PIPES_PREFIX)/lib \ -lhadooppipes -lhadooputils
为:
LDADD = -L$(HADOOP_UTILS_PREFIX)/lib -L$(HADOOP_PIPES_PREFIX)/lib \ -lhadooppipes -lhadooputils -lcrypto
即可.之后再重新运行ant -Dcompile.c++=yes examples即可编译通过。
更多技术文章、感悟、分享、勾搭,请用微信扫描:
相关推荐
hadoop-annotations-3.1.1.jar hadoop-common-3.1.1.jar hadoop-mapreduce-client-core-3.1.1.jar hadoop-yarn-api-3.1.1.jar hadoop-auth-3.1.1.jar hadoop-hdfs-3.1.1.jar hadoop-mapreduce-client-hs-3.1.1.jar ...
hadoop-mapreduce-examples-2.7.1.jar
hadoop2.6-common-bin 解决在Windows上操作hadoop出现 Could not locate executable问题
ambari-2.7.5 编译过程中四个大包下载很慢,所以需要提前下载,包含:hbase-2.0.2.3.1.4.0-315-bin.tar.gz ,hadoop-3.1.1.3.1.4.0-315.tar.gz , grafana-6.4.2.linux-amd64.tar.gz ,phoenix-5.0.0.3.1.4.0-315....
hadoop-mapreduce-examples-2.6.5.jar 官方案例源码
sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
Eclipse集成Hadoop2.10.0的插件,使用`ant`对hadoop的jar包进行打包并适应Eclipse加载,所以参数里有hadoop和eclipse的目录. 必须注意对于不同的hadoop版本,` HADDOP_INSTALL_PATH/share/hadoop/common/lib`下的jar包...
sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.zip
Hadoop第02天-03.完全分布式-搭建.avi!!
hadoop支持LZO压缩配置 将编译好后的hadoop-lzo-0.4.20.jar 放入hadoop-2.7.2/share/hadoop/common/ core-site.xml增加配置支持LZO压缩 <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl...
spark-1.6.3-bin-hadoop2.4-without-hive.tgz 经测试,hadoop 2.8.2下可用。hive2.1.1 可用
<copy file="${hadoop.root}/build/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core.jar" verbose="true"/> <copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli...
hadoop-2.6.0.tar.gz.mds,hadoop的安装包,版本为2.6.0,适应操作系统为Linux。
hadoop-2.7.3.tar.gz 下载 目前相对比较稳定的版本 hadoop tar 包下载hadoop-2.7.3.tar.gz 下载 hadoop tar 包下载
Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不了解分布式底层细节的情况下,开发分布式程序。充分利用集群的威力进 Hadoop是一个由Apache基金会所开发的分布式系统基础架构。用户可以在不...
sqoop-1.4.6.bin__hadoop-2.0.4-alpha.zip。。。。。。。。。。。。。。。。
hadoop-eclipse-plugin-2.7.1.jar插件,直接放在eclipse插件目录中
flink1.14.0与hadoop3.x的兼容包,放在flink的lib目录下
自己编译的64bithadoop-2.2.0版本 [INFO] Reactor Summary: ...This command was run using /home/hadoop/Desktop/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/share/hadoop/common/hadoop-common-2.2.0.jar