호롤롤롤롤롤롤롤롤롤롤롤로~ 멘붕이 오는구나 얼쑤 풍악을 울려라! 개객기 같은 아오
우분투 진짜 싫다 너무 싫다 진짜로 싫다.
- 해결만 보려면 맨밑.
Hadoop 멀티 환경을 구축했다. 버츄얼 말고 리얼로 4대 가지고.
OS는 Ubuntu 10.03이고
nameenode 1대
datanode 3대
start-dfs.sh 을 실행시킨다.
잘돌아간다!
namenode에서도 jps로 NN 뜨고 (세컨드NN도 뜨고)
datanode 들 에서도 jps로 DN가 뜬다! 잘 떠 !!
근데 !! hadoop dfsadmin -report를 하면 안나온다.. node도 뭐도.. 어떻게?? 이렇게
hadoop dfsadmin -report
Configured Capacity: 0 (0 GB)
Present Capacity: 0 (0 GB)
DFS Remaining: 0 (0 GB)
DFS Used: 0 (0 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
걍 안뜬다. 뭐지뭐지? 하다가 Datanode에서 log를 보니까 똭!
INFO org.apache.hadoop.ipc.RPC: Server at namenode1/(IP):(port) not available yet, Zzzzz...
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 0 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 1 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 2 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 3 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 4 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 5 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 6 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 7 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 8 time(s).
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode1/
(IP):(port) . Already tried 9 time(s).
이건 DN가 NN에게 연결을 못해서 시도하는 소리입니다..
그래서 Namenode에서 log를 보니까 똭!
INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9000: starting
INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9000: starting
아니 NN는 LISTEN 하고 있는데 서로들 삽질 중이다..
netstat -nlpt 해보니
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6738/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 723/cupsd
tcp6 0 0 :::41473 :::* LISTEN 17371/java
tcp6 0 0 127.0.1.1:9000 :::* LISTEN 17141/java
tcp6 0 0 :::50090 :::* LISTEN 17371/java
tcp6 0 0 :::56269 :::* LISTEN 17141/java
tcp6 0 0 :::50070 :::* LISTEN 17141/java
tcp6 0 0 :::22 :::* LISTEN 6738/sshd
tcp6 0 0 ::1:631 :::* LISTEN 723/cupsd
잘 돌아간다. 뭔지 몰라도 tcp6은 좀 수상해도... (IPv6인가..?) 다른 사람이 운영하는 Hadoop port 보니 tcp6인데 잘 돌아간다.
막 뭘까 고민하다가 namenode -format 할 때 자세히 봤다.
hadoop namenode -format
12/07/04 20:51:02 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = namenode1/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.3
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
************************************************************/
Re-format filesystem in /root/hdfs/name ? (Y or N) Y
12/07/04 20:51:05 INFO util.GSet: VM type = 32-bit
12/07/04 20:51:05 INFO util.GSet: 2% max memory = 8.89 MB
12/07/04 20:51:05 INFO util.GSet: capacity = 2^21 = 2097152 entries
12/07/04 20:51:05 INFO util.GSet: recommended=2097152, actual=2097152
12/07/04 20:51:06 INFO namenode.FSNamesystem: fsOwner=root
12/07/04 20:51:06 INFO namenode.FSNamesystem: supergroup=supergroup
12/07/04 20:51:06 INFO namenode.FSNamesystem: isPermissionEnabled=true
12/07/04 20:51:06 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
12/07/04 20:51:06 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
12/07/04 20:51:06 INFO namenode.NameNode: Caching file names occuring more than 10 times
12/07/04 20:51:06 INFO common.Storage: Image file of size 110 saved in 0 seconds.
12/07/04 20:51:07 INFO common.Storage: Storage directory /root/hdfs/name has been successfully formatted.
12/07/04 20:51:07 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at namenode1/127.0.1.1
************************************************************/
아래는 싱글 노드 환경에서 포맷했을때
hadoop namenode -format
12/07/04 20:51:02 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = namenode1/192.168.0.4
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.3
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by 'hortonfo' on Tue May 8 20:31:25 UTC 2012
************************************************************/
Re-format filesystem in /root/hdfs/name ? (Y or N) Y
12/07/04 20:51:05 INFO util.GSet: VM type = 32-bit
12/07/04 20:51:05 INFO util.GSet: 2% max memory = 8.89 MB
12/07/04 20:51:05 INFO util.GSet: capacity = 2^21 = 2097152 entries
12/07/04 20:51:05 INFO util.GSet: recommended=2097152, actual=2097152
12/07/04 20:51:06 INFO namenode.FSNamesystem: fsOwner=root
12/07/04 20:51:06 INFO namenode.FSNamesystem: supergroup=supergroup
12/07/04 20:51:06 INFO namenode.FSNamesystem: isPermissionEnabled=true
12/07/04 20:51:06 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
12/07/04 20:51:06 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
12/07/04 20:51:06 INFO namenode.NameNode: Caching file names occuring more than 10 times
12/07/04 20:51:06 INFO common.Storage: Image file of size 110 saved in 0 seconds.
12/07/04 20:51:07 INFO common.Storage: Storage directory /root/hdfs/name has been successfully formatted.
12/07/04 20:51:07 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at namenode1/192.168.0.4
************************************************************/
포맷 IP가 다르다? 뭐지 뭐지 하다가
/etc/hosts 파일을 보았다.
cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 namenode1 <- 윙?? 윙???????? 윙??????????????
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.0.6 datanode1
192.168.0.7 datanode2
192.168.0.8 datanode3
192.168.0.4 namenode1
127.0.1.1 은 뭐인가.. 127.0.0.1은 아는데 저 이상한 훼이크는 무엇인가..
지우고 재부팅을 했다.
다시 namenode -format
스압 관계상.. 생략하고 싱그모드로 포맷한 로그와 똑같이 뜬다.
IP가 정상으로 돌아왔다.
구동.. 해보니까 잘 돌아간다 ㅡㅡ
hadoop dfsadmin -report
Configured Capacity: 468544008192 (436.37 GB)
Present Capacity: 441155198976 (410.86 GB)
DFS Remaining: 441155141632 (410.86 GB)
DFS Used: 57344 (56 KB)
DFS Used%: 0%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
-------------------------------------------------
Datanodes available: 1 (1 total, 0 dead)
Name: 192.168.0.4:50010
Decommission Status : Normal
Configured Capacity: 468544008192 (436.37 GB)
DFS Used: 57344 (56 KB)
Non DFS Used: 27388809216 (25.51 GB)
DFS Remaining: 441155141632(410.86 GB)
DFS Used%: 0%
DFS Remaining%: 94.15%
Last contact: Wed Jul 04 20:51:28 KST 2012
아주 잘 된닼ㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋ 호롤롤롤롤롤롤롤롤롤로
진짜 우분투는 나랑 잘 안맞는다.. 물론 내가 우분투를 잘 안쓰고 공부를 안해서 나오는 불평이겠지만.
넷필터 방화벽 만들때도 우분투는 파tothe굌ㅋㅋㅋㅋㅋㅋㅋㅋㅋㅋ
아.. 필자의 지적 수준이 보이는 그런 잡글이였다. 나중에 고쳐야지 일단 안되는 분들을 위해 공개.
게이트 타고 해결만 보시려는 분을 위해 정리.
필자는 우분투를 씀.
멀티환경을 구축했음
datanode log에서 "블라블라 not available yet, Zzzzz..." 발견
서버는 리슨하고있고 클라는 접속을 요청중 근데 연결이 안됨.
/etc/hosts 밑에 수상한 127.0.1.1 localhost를 발견
네임노드 포맷할 때 host를 보니 namenode1/127.0.1.1로 저장됨.
namenode에서 netstat -nlpt해보니 오픈된 서버 ip가 127.0.1.1로 오픈됨.
자세히 보니까 127.0.0.1 이 아니고 127.0.1.1임.
/etc/hosts에서 127.0.1.1을 지우고 다시 포맷해봄.
잘됨
