Subscribed unsubscribe Subscribe Subscribe

CDH3u1→CDH3u2にアップデート

2011/10/21にCDH3u2がリリースされたのでアップデートしてみました。
CDH3 Installation Guide - Cloudera Support
Upgrading to CDH3 - Cloudera Support

$ hadoop version
Hadoop 0.20.2-cdh3u1
Subversion file:///tmp/nightly_2011-07-18_07-57-52_3/hadoop-0.20-0.20.2+923.97-1~maverick -r bdafb1dbffd0d5f2fbc6ee022e1c8df6500fd638
Compiled by root on Mon Jul 18 09:40:07 PDT 2011
From source with checksum 3127e3d410455d2bacbff7673bf3284c

現在はCDH3u1がインストールされてます。

$ for x in /etc/init.d/hadoop-* ; do sudo $x stop ; done
[sudo] password for h-akanuma: 
Stopping Hadoop datanode daemon: no datanode to stop
hadoop-0.20-datanode.
Stopping Hadoop jobtracker daemon: no jobtracker to stop
hadoop-0.20-jobtracker.
Stopping Hadoop namenode daemon: no namenode to stop
hadoop-0.20-namenode.
Stopping Hadoop secondarynamenode daemon: no secondarynamenode to stop
hadoop-0.20-secondarynamenode.
Stopping Hadoop tasktracker daemon: no tasktracker to stop
hadoop-0.20-tasktracker.
Stopping Hadoop HBase master daemon: no master to stop because kill -0 of pid 2271 failed with status 1
hbase-master.
Stopping Hadoop HBase regionserver daemon: stopping regionserver........
hbase-regionserver.
JMX enabled by default
Using config: /etc/zookeeper/zoo.cfg
Stopping zookeeper ... STOPPED
$ 
$ jps
9534 Jps
$ 
$ ps aux | grep hadoop
1000      9544  0.0  0.0   5164   788 pts/0    S+   21:56   0:00 grep --color=auto hadoop

Hadoop関連プロセスを停止。

$ sudo dpkg -i ダウンロード/cdh3-repository_1.0_all.deb 
未選択パッケージ cdh3-repository を選択しています。
(データベースを読み込んでいます ... 現在 262400 個のファイルとディレクトリがインストールされています。)
(.../cdh3-repository_1.0_all.deb から) cdh3-repository を展開しています...
cdh3-repository (1.0) を設定しています ...
gpg: 鍵輪「/etc/apt/secring.gpg」ができました
gpg: 鍵輪「/etc/apt/trusted.gpg.d/cloudera-cdh3.gpg」ができました
gpg: 鍵02A818DD: 公開鍵“Cloudera Apt Repository”を読み込みました
gpg: 処理数の合計: 1
gpg:               読込み: 1

ダウンロードしたパッケージをインストール

$ sudo apt-get update
・・・

APTパッケージインデックスを更新

$ apt-cache search hadoop
ubuntu-orchestra-modules-hadoop - Modules mainly used by orchestra-management-server
flume - reliable, scalable, and manageable distributed data collection application
hadoop-0.20 - A software platform for processing vast amounts of data
hadoop-0.20-conf-pseudo - Pseudo-distributed Hadoop configuration
hadoop-0.20-datanode - Data Node for Hadoop
hadoop-0.20-doc - Documentation for Hadoop
hadoop-0.20-fuse - HDFS exposed over a Filesystem in Userspace
hadoop-0.20-jobtracker - Job Tracker for Hadoop
hadoop-0.20-namenode - Name Node for Hadoop
hadoop-0.20-native - Native libraries for Hadoop (e.g., compression)
hadoop-0.20-pipes - Interface to author Hadoop MapReduce jobs in C++
hadoop-0.20-sbin - Server-side binaries necessary for secured Hadoop clusters
hadoop-0.20-secondarynamenode - Secondary Name Node for Hadoop
hadoop-0.20-source - Source code for Hadoop
hadoop-0.20-tasktracker - Task Tracker for Hadoop
hadoop-hbase - HBase is the Hadoop database
hadoop-hbase-doc - Documentation for HBase
hadoop-hbase-master - HMaster is the "master server" for a HBase
hadoop-hbase-regionserver - HRegionServer makes a set of HRegions available to clients
hadoop-hbase-thrift - Provides an HBase Thrift service
hadoop-hive - A data warehouse infrastructure built on top of Hadoop
hadoop-hive-metastore - Shared metadata repository for Hive
hadoop-hive-server - Provides a Hive Thrift service
hadoop-pig - A platform for analyzing large data sets using Hadoop
hadoop-zookeeper - A high-performance coordination service for distributed applications.
hadoop-zookeeper-server - This runs the zookeeper server on startup.
hue-common - A browser-based desktop interface for Hadoop
hue-filebrowser - A UI for the Hadoop Distributed File System (HDFS)
hue-jobbrowser - A UI for viewing Hadoop map-reduce jobs
hue-jobsub - A UI for designing and submitting map-reduce jobs to Hadoop
hue-plugins - Plug-ins for Hadoop to enable integration with Hue
hue-shell - A shell for console based Hadoop applications
libhdfs0 - JNI Bindings to access Hadoop HDFS from C
libhdfs0-dev - Development support for libhdfs0
mahout - A set of Java libraries for scalable machine learning.
oozie - A workflow and coordinator sytem for Hadoop jobs.
sqoop - Tool for easy imports and exports of data sets between databases and HDFS
cdh3-repository - Cloudera's Distribution including Apache Hadoop

Hadoopパッケージの検索

$ sudo apt-get install hadoop-0.20
・・・
$ hadoop version
Hadoop 0.20.2-cdh3u2
Subversion file:///tmp/nightly_2011-10-13_20-02-02_3/hadoop-0.20-0.20.2+923.142-1~maverick -r 95a824e4005b2a94fe1c11f1ef9db4c672ba43cb
Compiled by root on Thu Oct 13 21:52:18 PDT 2011
From source with checksum 644e5db6c59d45bca96cec7f220dda51

Hadoopコアパッケージをインストール。
CDH3u2がインストールされました。
Hadoop各デーモンも同時にアップデートされています。

$ sudo apt-get install hadoop-hbase-master
・・・
$ sudo apt-get install hadoop-zookeeper-server
・・・
$ hbase shell
11/10/26 22:36:54 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.90.4-cdh3u2, r, Thu Oct 13 20:32:26 PDT 2011

hbase(main):001:0> 

HBase, Zookeeper もアップデート。CDH3u2にアップデートされました。

$ sudo /etc/init.d/hadoop-0.20-namenode start
Starting Hadoop namenode daemon: starting namenode, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-namenode-h-akanuma-CF-W4.out
hadoop-0.20-namenode.
$ 
$ sudo /etc/init.d/hadoop-0.20-datanode start
Starting Hadoop datanode daemon: starting datanode, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-datanode-h-akanuma-CF-W4.out
hadoop-0.20-datanode.
$ 
$ sudo /etc/init.d/hadoop-0.20-secondarynamenode start
Starting Hadoop secondarynamenode daemon: starting secondarynamenode, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-secondarynamenode-h-akanuma-CF-W4.out
hadoop-0.20-secondarynamenode.
$ 
$ sudo /etc/init.d/hadoop-0.20-jobtracker start
Starting Hadoop jobtracker daemon: starting jobtracker, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-jobtracker-h-akanuma-CF-W4.out
hadoop-0.20-jobtracker.
$ 
$ sudo /etc/init.d/hadoop-0.20-tasktracker start
Starting Hadoop tasktracker daemon: starting tasktracker, logging to /usr/lib/hadoop-0.20/logs/hadoop-hadoop-tasktracker-h-akanuma-CF-W4.out
hadoop-0.20-tasktracker.
$ 
$ sudo jps
12799 SecondaryNameNode
12672 DataNode
12552 NameNode
12895 JobTracker
13029 Jps
11574 QuorumPeerMain
12996 TaskTracker

Hadoop各デーモンを起動

$ hadoop jar /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u2-*examples.jar pi 10 10000
Number of Maps  = 10
Samples per Map = 10000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
11/10/26 23:09:21 INFO mapred.FileInputFormat: Total input paths to process : 10
11/10/26 23:09:22 INFO mapred.JobClient: Running job: job_201110262307_0001
11/10/26 23:09:23 INFO mapred.JobClient:  map 0% reduce 0%
11/10/26 23:09:42 INFO mapred.JobClient:  map 20% reduce 0%
11/10/26 23:09:57 INFO mapred.JobClient:  map 40% reduce 0%
11/10/26 23:10:12 INFO mapred.JobClient:  map 60% reduce 0%
11/10/26 23:10:14 INFO mapred.JobClient:  map 60% reduce 13%
11/10/26 23:10:20 INFO mapred.JobClient:  map 80% reduce 20%
11/10/26 23:10:26 INFO mapred.JobClient:  map 100% reduce 20%
11/10/26 23:10:29 INFO mapred.JobClient:  map 100% reduce 33%
11/10/26 23:10:32 INFO mapred.JobClient:  map 100% reduce 100%
11/10/26 23:10:34 INFO mapred.JobClient: Job complete: job_201110262307_0001
11/10/26 23:10:35 INFO mapred.JobClient: Counters: 23
11/10/26 23:10:35 INFO mapred.JobClient:   Job Counters 
11/10/26 23:10:35 INFO mapred.JobClient:     Launched reduce tasks=1
11/10/26 23:10:35 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=113667
11/10/26 23:10:35 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
11/10/26 23:10:35 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
11/10/26 23:10:35 INFO mapred.JobClient:     Launched map tasks=10
11/10/26 23:10:35 INFO mapred.JobClient:     Data-local map tasks=10
11/10/26 23:10:35 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=49553
11/10/26 23:10:35 INFO mapred.JobClient:   FileSystemCounters
11/10/26 23:10:35 INFO mapred.JobClient:     FILE_BYTES_READ=226
11/10/26 23:10:35 INFO mapred.JobClient:     HDFS_BYTES_READ=2420
11/10/26 23:10:35 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=609632
11/10/26 23:10:35 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=215
11/10/26 23:10:35 INFO mapred.JobClient:   Map-Reduce Framework
11/10/26 23:10:35 INFO mapred.JobClient:     Reduce input groups=2
11/10/26 23:10:35 INFO mapred.JobClient:     Combine output records=0
11/10/26 23:10:35 INFO mapred.JobClient:     Map input records=10
11/10/26 23:10:35 INFO mapred.JobClient:     Reduce shuffle bytes=280
11/10/26 23:10:35 INFO mapred.JobClient:     Reduce output records=0
11/10/26 23:10:35 INFO mapred.JobClient:     Spilled Records=40
11/10/26 23:10:35 INFO mapred.JobClient:     Map output bytes=180
11/10/26 23:10:35 INFO mapred.JobClient:     Map input bytes=240
11/10/26 23:10:35 INFO mapred.JobClient:     Combine input records=0
11/10/26 23:10:35 INFO mapred.JobClient:     Map output records=20
11/10/26 23:10:35 INFO mapred.JobClient:     SPLIT_RAW_BYTES=1240
11/10/26 23:10:35 INFO mapred.JobClient:     Reduce input records=20
Job Finished in 74.586 seconds
Estimated value of Pi is 3.14120000000000000000

Hadoopジョブをテスト実行。
無事成功しました。

$ sudo /etc/init.d/hadoop-hbase-master start
Starting Hadoop HBase master daemon: starting master, logging to /usr/lib/hbase/logs/hbase-hbase-master-h-akanuma-CF-W4.out
hbase-master.
$ 
$ sudo /etc/init.d/hadoop-hbase-regionserver start
Starting Hadoop HBase regionserver daemon: starting regionserver, logging to /usr/lib/hbase/logs/hbase-hbase-regionserver-h-akanuma-CF-W4.out
hbase-regionserver.
$ 
$ sudo jps
14202 Jps
12799 SecondaryNameNode
12672 DataNode
14134 HRegionServer
13996 HMaster
12552 NameNode
12895 JobTracker
11574 QuorumPeerMain
12996 TaskTracker

HBaseのデーモンも起動。
擬似分散モードなのでZookeeperは起動させません。

$ hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.90.4-cdh3u2, r, Thu Oct 13 20:32:26 PDT 2011

hbase(main):001:0> 
hbase(main):002:0* list
TABLE                                                                                                            
courses                                                                                                          
scores                                                                                                           
2 row(s) in 2.0210 seconds

hbase(main):003:0> 

hbase shell の listコマンドで動作確認。
こちらも成功です。