Hadoopによるテキストマイニングの精度を上げる2つのアルゴリズム：テキストマイニングで始める実践Hadoop活用（最終回）（3/3 ページ）

 1 public class MultiClassLogisticRegressionMapper 
 2 extends Mapper<VIntWritable, MapWritable, VIntWritable, MapWritable> {
 3     private <MapVIntWritable, MapWritable> weightMap = new <HashMapVIntWritable, MapWritable>();
 4     private int classNum = 20;
 5     private int classWidth = 5;
 6     private double stepSize = 0.001;
 7     
 8     @Override
 9     public void setup(Context context) throws IOException, InterruptedException {
10         Configuration conf = context.getConfiguration();
11         stepSize = conf.getFloat("sgm.step.size", (float) stepSize);
12         classNum = conf.getInt("class.num", classNum);
13         classWidth = conf.getInt("class.width", classWidth);
14         String weightFile = conf.get("weight.file");
15         if (weightFile == null) {
16             for (int i = 0; i < classNum; i++) {
17                 weightMap.put(new VIntWritable(i), new MapWritable());
18             }
19         } else {
20             FileSystem fs = FileSystem.get(conf);
21             SequenceFile.Reader reader = new SequenceFile.Reader(fs, new Path(weightFile), conf);
22             try {
23                 VIntWritable key = new VIntWritable();
24                 MapWritable value = new MapWritable();
25                 while (reader.next(key, value)) {
26                     weightMap.put(new VIntWritable(key.get()), new MapWritable(value));
27                 }
28             } finally {
29                 reader.close();
30             }
31         }
32     }
33     @Override
34     public void map(VIntWritable key, MapWritable value, Context context)
35     throws IOException, InterruptedException {
36         int b = key.get();
37         value.put(new Text(CommonUtil.BIAS_KEY), new VIntWritable(1));
38         double[] y = new double[weightMap.size()];
39         double ySum = 0;
40         double wxMax = 0;
41         for (VIntWritable cat : weightMap.keySet()) {
42             double wx = DiscriminantFunctionAlgorithm.predict(value,weightMap.get(cat));
43             y[cat.get()] = wx;
44             wxMax = Math.max(wxMax, wx);
45         }
46         for (VIntWritable cat : weightMap.keySet()) {
47             double yk = Math.exp(y[cat.get()] - wxMax);
48             y[cat.get()] = yk;
49             ySum += yk;
50         }
51         for (VIntWritable cat : weightMap.keySet()) {
52             MapWritable weightMapPerCat = weightMap.get(cat);
53             int catInt = cat.get();
54             int bk = ((int) b / classWidth == catInt) ? 1 : 0;
55             double prob = y[catInt] / ySum;
56             for (Entry<Writable, Writable> entry : value.entrySet()) {
57                 Text word = (Text) entry.getKey();
58                 int x = ((VIntWritable) entry.getValue()).get();
59                 DoubleWritable weightWritable = (DoubleWritable) weightMapPerCat.get(word);
60                 double diffWeight = -stepSize * (prob - bk) * x;
61                 if (weightWritable == null) {
62                     weightMapPerCat.put(word, new DoubleWritable(diffWeight));
63                 } else {
64                     double w = weightWritable.get() + diffWeight;
65                     weightWritable.set(w);
66                 }
67             }
68         }
69     }
70     @Override
71     public void cleanup(Context context) 
72     throws IOException,InterruptedException {
73         for (VIntWritable key : weightMap.keySet()) {
74             MapWritable value = weightMap.get(key);
75             context.write(key, value);
76         }
77     }
78 }

public class MulticlassClassifierReducer 
extends Reducer<VIntWritable, MapWritable, VIntWritable, MapWritable> {
    @Override
    public void reduce(VIntWritable key, Iterable values,Context context) 
    throws IOException, InterruptedException {
……
    }
}

    private static void runIteration(Configuration conf, Path input, Path output)
    throws IOException, InterruptedException, ClassNotFoundException {
        Job job = new Job(conf, "logistic regression: " + output);
        job.setJarByClass(IterativeParameterMixingDriver.class);
        job.setMapperClass(MulticlassLogisticRegressionMapper.class);
        job.setReducerClass(MulticlassClassifierReducer.class);
        job.setInputFormatClass(SequenceFileInputFormat.class);
        job.setOutputFormatClass(SequenceFileOutputFormat.class);
        job.setOutputKeyClass(VIntWritable.class);
        job.setOutputValueClass(MapWritable.class);
        job.setNumReduceTasks(1);
        FileInputFormat.addInputPath(job, input);
        FileOutputFormat.setOutputPath(job, output);
        if (!job.waitForCompletion(true)) {
            throw new InterruptedException("job failed: " + output);
        }
    }

 1 public class TestMulticlassClassifier {
 2     private static int classWidth = 5;
 3     public static void main(String[] args) throws Exception {
 4         if (args.length < 2) {
 5             System.err.println("Usage: cmd <test file> <train file>");
 6             return;
 7         }
 8         MapWritable testMap = CommonUtil.readTestFile(args[0]);
 9         Map<VIntWritable, MapWritable> weightMap = CommonUtil.readMulticlassWeightFile(args[1]);
10         testMap.put(new Text(CommonUtil.BIAS_KEY), new VIntWritable(1));
11         int maxCat = 0;
12         double maxYk = 0;
13         for (VIntWritable cat : weightMap.keySet()) {
14             double yk = DiscriminantFunctionAlgorithm.predict(testMap,
15             weightMap.get(cat));
16             if (yk > maxYk) {
17                 maxYk = yk;
18                 maxCat = cat.get();
19             }
20         }
21         int predict = maxCat * classWidth;
22         System.out.println("" + predict + "" + (predict + 5) + "");
23     }
24 }