Sep
18
Trying out mapreduce - on the farm
Received an email this week from Sanger helpdesk that they installed a test hadoop system on the farm with 2 nodes. Thanks guys! First thing to do, obviously, was to repeat the streaming mapreduce exercise I did on my own machine (see my previous post). Only difference with my local setup is that this time I had to handle HDFS.
As a recap from my previous post: I'll be running an equivalent of the following:
cat snps.txt | ruby snp_mapper.rb | sort | ruby snp_reducer.rb
Setting up
First thing the Sanger wiki told me was to format my HDFS space:hadoop namenode -format This apparently only affects my own space... After that I could start playing with hadoop.
As a recap from my previous post: I'll be running an equivalent of the following:
cat snps.txt | ruby snp_mapper.rb | sort | ruby snp_reducer.rb
Setting up
First thing the Sanger wiki told me was to format my HDFS space:hadoop namenode -format This apparently only affects my own space... After that I could start playing with hadoop.
