本博客属原创文章,转载请注明出处:http://guoyunsky.iteye.com/blog/1245092
欢迎加入Hadoop超级群: 180941958
Oozie下面有很多例子,提供测试.也可以将源码放入Eclipse中启动提交.这里就一起试下.但发现有些问题,一一解决吧.
运行Oozie例子map-reduce,命令:
$OOZIE_HOME/bin/oozie job -oozie http://localhost:11000/oozie -config /home/guoyun/hadoop/oozie-3.0.2/examples/apps/map-reduce/job.properties -run
本以为会很顺利,但控制台报以下错误:
Error: E0902 : E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
跑到日志中$OOZIE_HOME/logs/oozie.log发现有以下报错:
2011-11-09 10:42:30,721 WARN V1JobsServlet:539 - USER[?] GROUP[users] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v1/jobs?action=start] error[E0902], E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
org.apache.oozie.servlet.XServletException: E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:196)
at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:89)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:281)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.service.AuthorizationException: E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:320)
at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:185)
... 16 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
at org.apache.oozie.service.KerberosHadoopAccessorService.createFileSystem(KerberosHadoopAccessorService.java:208)
at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:285)
... 17 more
Caused by: java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1131)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at $Proxy22.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:398)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:384)
at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:111)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:213)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:180)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1514)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1548)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1530)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:200)
at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:192)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
at org.apache.oozie.service.KerberosHadoopAccessorService.createFileSystem(KerberosHadoopAccessorService.java:192)
... 18 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:425)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:532)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:210)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1244)
at org.apache.hadoop.ipc.Client.call(Client.java:1075)
... 37 more
从Oozie的FAQ找到了解决办法:https://github.com/yahoo/oozie/wiki/FAQ,需要将oozie-default.xml中的 oozie.services属性拷贝到oozie-site.xml中,也就是以下属性:
<property>
<name>oozie.services</name>
<value>
org.apache.oozie.service.SchedulerService,
org.apache.oozie.service.InstrumentationService,
org.apache.oozie.service.CallableQueueService,
org.apache.oozie.service.UUIDService,
org.apache.oozie.service.ELService,
org.apache.oozie.service.AuthorizationService,
org.apache.oozie.service.KerberosHadoopAccessorService,
org.apache.oozie.service.MemoryLocksService,
org.apache.oozie.service.DagXLogInfoService,
org.apache.oozie.service.SchemaService,
org.apache.oozie.service.LiteWorkflowAppService,
org.apache.oozie.service.JPAService,
org.apache.oozie.service.StoreService,
org.apache.oozie.service.CoordinatorStoreService,
org.apache.oozie.service.SLAStoreService,
org.apache.oozie.service.DBLiteWorkflowStoreService,
org.apache.oozie.service.CallbackService,
org.apache.oozie.service.ActionService,
org.apache.oozie.service.ActionCheckerService,
org.apache.oozie.service.RecoveryService,
org.apache.oozie.service.PurgeService,
org.apache.oozie.service.CoordinatorEngineService,
org.apache.oozie.service.BundleEngineService,
org.apache.oozie.service.DagEngineService,
org.apache.oozie.service.CoordMaterializeTriggerService,
org.apache.oozie.service.StatusTransitService,
org.apache.oozie.service.PauseTransitService
</value>
<description>
All services to be created and managed by Oozie Services singleton.
Class names must be separated by commas.
</description>
</property>
拷贝完成后重启OOZIE,继续跑map-reduce例子,控制台上却又报一样的错误:
Error: E0902 : E0902: Exception occured: [java.net.ConnectException: Call to localhost/127.0.0.1:8020 failed on connection exception: java.net.ConnectException: Connection refused]
nameNode=hdfs://localhost:9000
jobTracker=localhost:9001
queueName=default
examplesRoot=hadoop/oozie-3.0.2/examples
oozie.coord.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/aggregator
outputDir=map-reduce
于是继续运行这个example,报如下错误:
Error: E0504 : E0504: App directory [hdfs://localhost:9000/user/guoyun/hadoop/oozie-3.0.2/examples/apps/aggregator] does not exist
想想我现在是伪分布式下运行Hadoop,从HDFS中拿文件则需要先存入到HDFS中.还是直接调用本地文件方便调试,于是将job.properties修改如下:
nameNode=file://
jobTracker=localhost:9001
queueName=default
examplesRoot=hadoop/oozie-3.0.2/examples
oozie.wf.application.path=${nameNode}/home/${user.name}/${examplesRoot}/apps/map-reduce
outputDir=map-reduce
再次运行得到以下结果:
job: 0000001-111109111632287-oozie-guoy-W
再去http://localhost:11000/oozie查看也能看到该job,并有相应的状态.
OK!example运行成功!
更多技术文章、感悟、分享、勾搭,请用微信扫描:
相关推荐
Hadoop学习笔记,自己总结的一些Hadoop学习笔记,比较简单。
Hadoop 学习笔记.md
hadoop学习笔记,hadoop简介,适用于hadoop入门,讲解hadoop安装,使用,基本原理,大数据,分布式等概念
云计算,hadoop,学习笔记, dd
hadoop 学习笔记,从搭建环境开始到具体实验。包括hdfs配置,yarn配置,分布式配置,如何编写mapreuduce 一步一步手把手,最后项目是hadoop 与 javaweb
大数据平台,hadoop集群学习笔记
踏入hadoop的世界,一个不一样的世界
Hadoop oozie报错:Table ‘oozie.VALIDATE_CONN’ doesn’t exist 。 oozie建表sql,直接下载执行
Hadoop架构分析之集群结构分析,Hadoop架构分析之HDFS架构分析,Hadoop架构分析之NN和DN原生文档解读,Hadoop MapReduce原理之流程图.Hadoop MapReduce原理之核心类Job和ResourceManager解读.Hadoop MapReduce原理之...
我学习hadoop的笔记,并在公司做的报告,给大家共享下
Hadoop学习笔记
hadoop学习笔记.rarhadoop学习笔记.rarhadoop学习笔记.rarhadoop学习笔记.rarhadoop学习笔记.rarhadoop学习笔记.rarhadoop学习笔记.rar
使用命令行编译打包运行自己的MapReduce程序 Hadoop2.6.0
Hadoop学习笔记AAAAAAAAAAA
hadoop oozie启动或运行报错:Table ‘oozie.VALIDATE_CONN’ doesn’t exist
Oozie由Cloudera公司贡献给Apache的基于工作流引擎的开源框架,是用于Hadoop平台的开源的工作流调度引擎,是用来管理Hadoop作业,属于web应用程序,由Oozie client和Oozie Server两个组件构成,Oozie Server运行于Java ...
第4章 Oozie的使用 4.1 案例一:Oozie调度shell脚本 目标:使用Oozie调度Shell脚本 分步实现: 1)解压官方案例模板 [atguigu@hadoop102 oozie-4.0.0-cdh5.3.6]$ tar -zxvf oozie-examples.tar.gz 2)创建工作目录 ...
hadoop集群安装笔记,我个人在自己的本机虚拟3台机器,搭建hadoop的学习环境,在安装过程中,出现这样那样的问题,并记录下来分享给大家,希望都新手有帮助。
大数据之Hadoop学习教程+笔记合计_超详细完整.zip