Posted on September 29, 2015 12:59 pm
 |  Asked by Roger Wilco
 |  411 views
RESOLVED
0
0
Print Friendly, PDF & Email

I’ve been trying to test out the Hadoop mapred tracer feature in vEOS 4.15.0F, and the active jobs, history,  and tasktracker never reflects the currently running jobs. I set up a two machine Hadoop 1.2.1 cluster using Debian Wheezy/Jessie (everything at defaults) following the User Manual,  and while the commands take, nothing seems to happen on the arista side. There’s full connectivity between the hadoop servers, and a little Jessie “management station”. I can see the running jobs just fine via the tasktracker http page and logs. I even tried using multiple Oracle and Openjdk versions, but it’s always the same.

Am I missing something obvious here?

0
Posted by Richard
Answered on September 29, 2015 3:28 pm

Please provide the full MapReduce Tracer config, the output of ”show monitor hadoop status” and ”show monitor hadoop cluster all”.

Thanks

0
Posted by Roger Wilco
Answered on October 1, 2015 5:41 pm

Switch(config)#sh run
…cut…
monitor hadoop
no shutdown
cluster CL1
jobtracker host 10.10.10.10 rpc-port 54311 user hduser
no shutdown
…cut…

For whatever reason, tasktracker http-port 50060 doesn’t appear in the running config, even though I set it up right after configuring jobtracker.
Switch(config-monitor-hadoop-CL1)#sh monitor hadoop status
Last updated: 2015-10-01 17:30:32
Mapreduce Tracer status:
Admin status : Enabled
Operational status : Enabled
Number of clusters configured : 1
Number of local TaskTrackers : 0
Number of jobs running locally : 0
Cluster Errors : None
Switch(config-monitor-hadoop-CL1)#sh monitor hadoop cluster all
Total number of clusters configured: 1
Cluster : CL1
Admin status : Enabled
JobTracker host : 10.10.10.10
JobTracker RPC port : 54311
JobTracker user : hduser
JobTracker polling interval : 10 seconds
TaskTracker HTTP port : 50060
Operational status : Enabled
Active TaskTrackers : 2
Blacklisted TaskTrackers : 0
Decommissioned TaskTrackers : 0
Tracker expiry interval : 600000.0
Map slots (used/total) : 0/4
Reduce slots (used/total) : 0/4
JobTracker heap size : 15.19MB (max: 966.69MB)

0
Posted by Gareth
Answered on October 2, 2015 11:16 am

Roger,

Here is a config from one of the leaf switches I used when developing the MapReduce Tracer TOI (I think from the user names etc, this is probably your source material). Important elements include the SVI on the server VLAN (source for MapReduce Tracer messages) and the ip host info for all Hadoop hosts.

! Command: show running-config
! device: vEOS-L1 (vEOS, EOS-4.13.0-1672898.4133F.1 (engineering build))
!
! boot system flash:/vEOS.swi
!
transceiver qsfp default-mode 4x10G
!
hostname vEOS-L1
ip host hadoop101 192.168.10.1
ip host hadoop102 192.168.10.2
ip host hadoop103 192.168.10.3
ip host hadoop201 192.168.20.1
ip host hadoop202 192.168.20.2
ip host hadoop203 192.168.20.3
!
spanning-tree mode mstp
!
vlan 10
!
interface Ethernet1
description P2P-L2 link to vEOS-S1
no switchport
ip address 192.168.11.2/24
!
interface Ethernet2
description Interface for hadoop101
switchport access vlan 10
!
interface Ethernet3
description Interface for hadoop102
switchport access vlan 10
!
interface Ethernet4
description Interface for hadoop103
switchport access vlan 10
!
interface Loopback0
description RouterID for vEOS-L1
ip address 10.0.0.21/32
!
interface Management1
ip address 192.168.9.21/24
no lldp transmit
no lldp receive
!
interface Vlan10
ip address 192.168.10.254/24
!
interface Vlan100
ip address 192.168.100.1/24
!
monitor hadoop
no shutdown
cluster VLAB
jobtracker host hadoop101 rpc-port 54311 user hduser
interval 5
no shutdown
!
ip route 0.0.0.0/0 192.168.9.254
!
ip routing
!
router ospf 1
router-id 10.0.0.21
network 192.168.10.0/24 area 0.0.0.0
network 192.168.11.0/24 area 0.0.0.0
network 192.168.100.0/24 area 0.0.0.0
max-lsa 12000
maximum-paths 16

!
end

0
Posted by Roger Wilco
Answered on October 2, 2015 11:49 pm

I couldn’t read the Hadoop TOI, since when I searched for anything Hadoop related here, the articles all say I lack permissions to read them. I ended up following Michael Noll’s setup example from here.

I reconfigured the switch following the config you posted (I only changed the IPs, and used 2 hadoop servers), but the stats still don’t reflect the currently running jobs:

SW(config-monitor-hadoop)#show monitor hadoop cluster CL1 tasktracker
Last updated: 2015-10-02 23:41:17
Total 2 TaskTrackers on cluster CL1:
Node IP Address Interface State Maps Reduces
———— —————– ————— ———— ———- ——-
master 10.10.10.10 Ethernet1 active 0 0
slave 10.10.10.11 Ethernet2 active 0 0

SW(config-monitor-hadoop)#show monitor hadoop cluster CL1 tasktracker status
Last updated: 2015-10-02 23:41:22
Total 2 TaskTrackers on cluster CL1:
TaskTracker : slave
IP Address : 10.10.10.11
Interface : Ethernet2
State : active
State detail : ARP and DNS are resolved
Running jobs : 0
Running tasks : 0
Map Tasks : 0
Reduce Tasks : 0
Total bytes read : 0
Total bytes written : 0

TaskTracker : master
IP Address : 10.10.10.10
Interface : Ethernet1
State : active
State detail : ARP and DNS are resolved
Running jobs : 0
Running tasks : 0
Map Tasks : 0
Reduce Tasks : 0
Total bytes read : 0
Total bytes written : 0

SW(config-monitor-hadoop)#show monitor hadoop cluster CL1 history
No history found
SW(config-monitor-hadoop)#show monitor hadoop cluster CL1 jobs
No running jobs found
SW(config-monitor-hadoop)#

0
Posted by Richard
Answered on October 5, 2015 4:02 pm

MapReduce Tracer resolves host names learned from the Hadoop infrastructure. Based on this, you need either host entries or DNS to resolve Hadoop nodenames.

ip name-server # This assumes Hadoop nodes are resolvable by this server
OR
ip host

Let us know if this takes care of your issue.

0
Posted by Richard
Answered on October 5, 2015 7:32 pm

Please disregard my last post, I missed this output ”State detail : ARP and DNS are resolved”

I think your best course of action at this time is to contact our support line at 866-476-0000 so that we may involve all necessary resources to resolve your issue.

0
Posted by Roger Wilco
Answered on October 5, 2015 8:25 pm

I added host entries to the switch, and the hadoop servers, and they can ping each other via hostnames.  Still no change.

Post your Answer

You must be logged in to post an answer.