问题:

1.在读取配置文件时，无法找到mapreduce.outputformat.class 的对应值

12/12/12 17:30:11 ERROR loader.OraLoader: mapreduce.outputformat.class is invalid

oracle.hadoop.loader.OraLoaderException: missing or invalid property "mapreduce.outputformat.class" value

解决：在执行hadoop   ${OLH_JAR}    oracle.hadoop.loader.OraLoader -conf /home/oracle/OLH/MyConf.xml 时，将MyConf.xml文件的路径写成了 HDFS文件系统的路径，将其改为LINUX 下的路径。

 

2.在进行连接时，无法连接监听器

解决:  没有配置好，本机的lisener,tnsname ,sqlnet 文件

 

3.在进行Loader时需要的文件， IN_DIRECTORY,OUT_DIRECTORY, DATAFILE,

重要的对象:

HDFS:

start:

mapred.input.dir,

   mapreduce.outputformat.class

   mapreduce.inputformat.class

  mapreduce.output.dir

end:

 oracle.hadoop.loader.loaderMapFile

oracle.hadoop.loader.targetTable

oracle.hadoop.loader.input.fieldNames

oracle.hadoop.loader.connection.url

 

 

 

<

name

>

mapreduce.inputformat.class

</

name

>

 

<

value

>

oracle.hadoop.loader.lib.input.DelimitedTextInputFormat

</

value

>

 

 

<

name

>

mapred.input.dir

</

name

>

 

<

value

>

/user/root/olh_lab_in/data.dat

</

value

>

 

 

 

<

name

>

mapreduce.outputformat.class

</

name

>

 

<

value

>

oracle.hadoop.loader.lib.output.JDBCOutputFormat

</

value

>

 

<

name

>

mapred.output.dir

</

name

>

 

<

value

>

/user/root/olh_lab_out

</

value

>

 

 

 

<

name

>

oracle.hadoop.loader.loaderMapFile

</

name

>

 

<

value

>

file:///home/oracle/OLH/loaderMap_exercise1.xml

</

value

>

    

<name>oracle.hadoop.loader.targetTable</name>

<value>SCOTT.OLH_TABLE</value>

 

<name>oracle.hadoop.loader.input.fieldNames</name>

<value>col1,col2,col3,col4</value>

  

 

<

name

>

oracle.hadoop.loader.connection.url

</

name

>

 

<

value

>

jdbc:oracle:thin:@${HOST}:${TCPPORT}/${SERVICE_NAME}

</

value

>

 

<

name

>

TCPPORT

</

name

>

 

<

value

>

1521

</

value

>

 

<

name

>

HOST

</

name

>

 

<

value

>

hadoop.oracle

</

value

>

 

<

name

>

ORACLE_SID

</

name

>

 

<

value

>

hadoop

</

value

>

 

<

name

>

SERVICE_NAME

</

name

>

 

<

value

>

hadoop

</

value

>

 

<

name

>

oracle.hadoop.loader.connection.user

</

name

>

 

<

value

>

SCOTT

</

value

>

 

<

description

>

AvroSchemaAwareDBInputFormat wants this case sensitive! (cause mapred.jdbc.username points here)

</

description

>

 

<

name

>

oracle.hadoop.loader.connection.password

</

name

>

 

<

value

>

tiger

</

value

>

 

 

 

 

[oracle@hadoop admin]$ hadoop dfs -rmr /user/root/olh_lab_out

Deleted hdfs://hdnode1m:9000/user/root/olh_lab_out

[oracle@hadoop admin]$ hadoop jar $OLH_HOME/jlib/oraloader.jar oracle.hadoop.loader.OraLoader -conf /home/oracle/OLH/MyConf.xml

Oracle Loader for Hadoop Release 2.0.0 - Production

 

Copyright (c) 2011, 2012, Oracle and/or its affiliates. All rights reserved.

 

13/03/07 19:38:38 INFO loader.OraLoader: Oracle Loader for Hadoop Release 2.0.0 - Production

 

Copyright (c) 2011, 2012, Oracle and/or its affiliates. All rights reserved.

 

13/03/07 19:38:38 INFO loader.OraLoader: Built-Against: hadoop-0.20.2 hive-0.7.1-cdh3u3 avro-1.6.3 jackson-1.8.8

13/03/07 19:38:39 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=

13/03/07 19:38:40 INFO loader.OraLoader: oracle.hadoop.loader.loadByPartition is disabled because table: OLH_TABLE is not partitioned

13/03/07 19:38:40 INFO loader.OraLoader: oracle.hadoop.loader.enableSorting disabled, no sorting key provided

13/03/07 19:38:40 INFO output.DBOutputFormat: Setting reduce tasks speculative execution to false for : oracle.hadoop.loader.lib.output.JDBCOutputFormat

13/03/07 19:38:40 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized

13/03/07 19:38:40 WARN loader.OraLoader: Sampler error: the number of reduce tasks must be greater than one; the configured value is 1 . Job will continue without sampled information.

13/03/07 19:38:40 INFO loader.OraLoader: Sampling time=0D:0h:0m:0s:13ms (13 ms)

13/03/07 19:38:40 INFO loader.OraLoader: Submitting OraLoader job OraLoader

13/03/07 19:38:42 INFO input.FileInputFormat: Total input paths to process : 1

13/03/07 19:38:42 INFO input.FileInputFormat: Total input paths to process : 1

13/03/07 19:38:42 INFO mapred.MapTask: io.sort.mb = 100

13/03/07 19:38:43 INFO mapred.MapTask: data buffer = 79691776/99614720

13/03/07 19:38:43 INFO mapred.MapTask: record buffer = 262144/327680

13/03/07 19:38:43 INFO mapred.MapTask: Starting flush of map output

13/03/07 19:38:43 INFO mapred.MapTask: Finished spill 0

13/03/07 19:38:43 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting

13/03/07 19:38:43 INFO mapred.LocalJobRunner:

13/03/07 19:38:43 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done.

13/03/07 19:38:43 INFO mapred.LocalJobRunner:

13/03/07 19:38:43 INFO mapred.Merger: Merging 1 sorted segments

13/03/07 19:38:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 1196 bytes

13/03/07 19:38:43 INFO mapred.LocalJobRunner:

13/03/07 19:38:43 INFO output.DBOutputFormat: conf prop: defaultExecuteBatch: 100

13/03/07 19:38:43 INFO output.DBOutputFormat: conf prop: loadByPartition: false

13/03/07 19:38:43 INFO output.DBOutputFormat: Insert statement: INSERT INTO "SCOTT"."OLH_TABLE" ("COL1", "COL2", "COL3", "COL4") VALUES (?, ?, ?, ?)

13/03/07 19:38:43 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting

13/03/07 19:38:43 INFO mapred.LocalJobRunner:

13/03/07 19:38:43 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now

13/03/07 19:38:43 INFO output.JDBCOutputFormat: Committed work for task attempt attempt_local_0001_r_000000_0

13/03/07 19:38:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to /user/root/olh_lab_out

13/03/07 19:38:43 INFO mapred.LocalJobRunner: reduce > reduce

13/03/07 19:38:43 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done.

13/03/07 19:38:43 INFO loader.OraLoader: map 100% reduce 100%

13/03/07 19:38:43 INFO loader.OraLoader: Job complete: OraLoader (null)

13/03/07 19:38:43 INFO loader.OraLoader: Counters: 14

        FileSystemCounters

                FILE_BYTES_READ=18885058

                FILE_BYTES_WRITTEN=55486

                HDFS_BYTES_READ=82429

                HDFS_BYTES_WRITTEN=18957078

        Map-Reduce Framework

                Combine input records=0

                Combine output records=0

                Map input records=20

                Map output bytes=1154

                Map output records=20

                Reduce input groups=1

                Reduce input records=20

                Reduce output records=20

                Reduce shuffle bytes=0

                Spilled Records=40