Convolutional Neural Nets

Instructions:

  1. Implement a convolutional layer for your neural network. To make this task simple, a Tensor class with a convolution method was included in the starter kit. The constructor for a convolutional layer might look like this:

    C++:
    size_t countTensorElements(const std::initializer_list<size_t>& dims)
    {
    	size_t n = 1;
    	for(const size_t* it = begin(dims); it != end(dims); ++it)
    		n *= *it;
    	return n;
    }
    
    LayerConv::LayerConv(const std::initializer_list<size_t>& inputDims,
                         const std::initializer_list<size_t>& filterDims,
                         const std::initializer_list<size_t>& outputDims)
    : Layer(countTensorElements(inputDims), countTensorElements(outputDims))
    {
    	...
    


    Java:
    	LayerConv(int[] inputDims, int[] filterDims, int[] outputDims)
    	{
    		...
    


    Such layers could be instantiated like this:

    C++:
    		nn.add(new LayerConv({28, 28}, {5, 5, 32}, {28, 28, 32}));
    


    Java:
    		nn.add(new LayerConv(new int[]{28, 28},
    		                     new int[]{5, 5, 32},
    		                     new int[]{28, 28, 32}));
    


  2. Implement a 2D max pooling layer.


  3. Add a unit test that uses finite differencing to validate that your implementation computes correct gradients for the following topology:
    nn.add(new LayerConv({8, 8}, {5, 5, 4}, {8, 8, 4}));
    nn.add(new LayerLeakyRectifier(8 * 8 * 4));
    nn.add(new LayerMaxPooling2D(8, 8, 4));
    nn.add(new LayerConv({4, 4, 4}, {3, 3, 4, 6}, {4, 4, 1, 6}));
    nn.add(new LayerLeakyRectifier(4 * 4 * 6));
    nn.add(new LayerMaxPooling2D(4, 4, 1 * 6));
    nn.add(new LayerLinear(2 * 2 * 6, 3));
    


  4. (Optional) The following topology will yield fewer than 60 errors on the MNIST dataset after about 2 or 3 days of training. If you have access to a machine on which you can run for that long, it might be fun to test your code.
    nn.add(new LayerConv({28, 28}, {5, 5, 32}, {28, 28, 32}));
    nn.add(new LayerLeakyRectifier(28 * 28 * 32));
    nn.add(new LayerMaxPooling2D(28, 28, 32));
    nn.add(new LayerConv({14, 14, 32}, {5, 5, 32, 64}, {14, 14, 1, 64}));
    nn.add(new LayerLeakyRectifier(14 * 14 * 64));
    nn.add(new LayerMaxPooling2D(14, 14, 64));
    nn.add(new LayerLinear(7 * 7 * 64, 1000));
    nn.add(new LayerLeakyRectifier(1000));
    nn.add(new LayerLinear(1000, 10));
    nn.add(new LayerLeakyRectifier(10));
    
    (If you do this, that's awesome, but please don't submit code to me that does it.)


  5. When I execute your code, it should run your finite differencing test. It should print the empirical gradients and the computed gradients to the console, and it should print some message to indicate whether the test passed. Submit your code in the usual manner.


FAQ:

  1. The Tensor.convolve method requires the filters to have the same number of dimensions as the input. How do we implement it to support larger filters?
    You will need to use a loop. For example, if the input has a size of {28, 28} and the filter has a size of {5, 5, 32}, then you will need to call Tensor.convolve 32 times (each time using a {5,5} filter) to complete the convolution.


  2. Do convolutional layers have bias terms?
    Yes, there is one bias value for each filter. It is added to each of the output values.


  3. How does one initialize the weights of a convolutional layer?
    I initialize each weight to "r/c", where "r" is a random value drawn from a normal distribution, and "c" is the total number of elements in the filter.


  4. Q: Can you provide example debug spew to help me debug it?
    A: Okay.
    
    I trained a 3-layer neural network with this topology:
    	Layer 0: Convolution({4, 4}, {3, 3, 2}, {4, 4, 2})
    	Layer 1: LeakyRectifier(4 * 4 * 2)
    	Layer 2: MaxPooling2D(4, 4, 2)
    
    Total number of weights: 20
    	(3*3+1)*2 = 20
    	The "+1" is for the bias term.
    	There is one bias value for each of the 2 filters.
    
    initial weights:
    	0,              (bias #1)
    	0.01,0.02,0.03, (filter #1)
    	0.04,0.05,0.06,
    	0.07,0.08,0.09,
    
    	0.1,            (bias #2)
    	0.11,0.12,0.13, (filter #2)
    	0.14,0.15,0.16,
    	0.17,0.18,0.19
    
    input vector:
    	0,0.1,0.2,0.3,
    	0.4,0.5,0.6,0.7,
    	0.8,0.9,1,1.1,
    	1.2,1.3,1.4,1.5
    
    Layer 0 activation:
    	0.083,0.139,0.178,0.121,
    	0.198,0.303,0.348,0.225,
    	0.33,0.483,0.528,0.333,
    	0.181,0.253,0.274,0.163,
    
    	0.283,0.419,0.518,0.401,
    	0.568,0.853,0.988,0.715,
    	0.94,1.393,1.528,1.063,
    	0.701,1.013,1.094,0.763
    
    Layer 1 activation:
    	0.083,0.139,0.178,0.121,
    	0.198,0.303,0.348,0.225,
    	0.33,0.483,0.528,0.333,
    	0.181,0.253,0.274,0.163,
    
    	0.283,0.419,0.518,0.401,
    	0.568,0.853,0.988,0.715,
    	0.94,1.393,1.528,1.063,
    	0.701,1.013,1.094,0.763
    
    	(Leaky rectifier did nothing because all values were positive.)
    
    Layer 2 activation:
    	0.303,0.348,
    	0.483,0.528,
    
    	0.853,0.988,
    	1.393,1.528
    
    target:
    	0.7,0.6,
    	0.5,0.4,
    
    	0.3,0.2,
    	0.1,0
    
    Layer 2 blame:
    	0.397,0.252,
    	0.017,-0.128,
    
    	-0.553,-0.788,
    	-1.293,-1.528
    
    Layer 1 blame:
    	0,0,0,0,
    	0,0.397,0.252,0,
    	0,0.017,-0.128,0,
    	0,0,0,0,
    
    	0,0,0,0,
    	0,-0.553,-0.788,0,
    	0,-1.293,-1.528,0,
    	0,0,0,0
    
    Layer 0 blame:
    	0,0,0,0,
    	0,0.397,0.252,0,
    	0,0.017,-0.128,0,
    	0,0,0,0,
    
    	0,0,0,0,
    	0,-0.553,-0.788,0,
    	0,-1.293,-1.528,0,
    	0,0,0,0
    
    gradient:
    	0.538,
    	-0.032,0.0218,0.0756,
    	0.1832,0.237,0.2908,
    	0.3984,0.4522,0.506,
    
    	-4.162,
    	-1.36,-1.7762,-2.1924,
    	-3.0248,-3.441,-3.8572,
    	-4.6896,-5.1058,-5.522
    
    updated weights:
    	0.00538,
    	0.00968,0.020218,0.030756,
    	0.041832,0.05237,0.062908,
    	0.073984,0.084522,0.09506,
    
    	0.05838,
    	0.0964,0.102238,0.108076,
    	0.109752,0.11559,0.121428,
    	0.123104,0.128942,0.13478
    


  5. Q: Can you please add another convolutional layer to the spew to help us debug the backprop step?
    A: Yes.
    I trained a 4-layer neural network with this topology:
    	Layer 0: Convolution({4, 4}, {3, 3}, {4, 4})
    	Layer 1: Convolution({4, 4}, {3, 3, 2}, {4, 4, 2})
    	Layer 2: LeakyRectifier(4 * 4 * 2)
    	Layer 3: MaxPooling2D(4, 4, 2)
    
    Total number of weights: 30
    	3*3+1 + (3*3+1)*2 = 20
    	The "+1" is for the bias term.
    	There is one bias value for the filter in layer 0,
    	and one biasvalue for each of the 2 filters in layer 1.
    
    initial weights:
    	0,              (bias in layer 0)
    	0.01,0.02,0.03, (filter in layer 0)
    	0.04,0.05,0.06,
    	0.07,0.08,0.09,
    
    	0.1,            (bias for first filter in layer 1)
    	0.11,0.12,0.13, (first filter in layer 1)
    	0.14,0.15,0.16,
    	0.17,0.18,0.19
    
    	0.20,           (bias for second filter in layer 1)
    	0.21,0.22,0.23, (second filter in layer 1)
    	0.24,0.25,0.26,
    	0.27,0.28,0.29,
    
    
    input vector:
    	0,0.1,0.2,0.3,
    	0.4,0.5,0.6,0.7,
    	0.8,0.9,1,1.1,
    	1.2,1.3,1.4,1.5
    
    Layer 0 activation: 
    	0.083,0.139,0.178,0.121,
    	0.198,0.303,0.348,0.225,
    	0.33,0.483,0.528,0.333,
    	0.181,0.253,0.274,0.163
    
    Layer 1 activation: 
    	0.2279,0.31527,0.32242,0.24273,
    	0.35738,0.52116,0.52342,0.36627,
    	0.37058,0.53488,0.52774,0.36507,
    	0.27002,0.37003,0.36238,0.26085,
    
    	0.4002,0.54017,0.55382,0.42993,
    	0.61098,0.88016,0.88922,0.63957,
    	0.64538,0.92468,0.91874,0.65217,
    	0.49472,0.67493,0.66578,0.49065
    
    Layer 2 activation:
    	0.2279,0.31527,0.32242,0.24273,
    	0.35738,0.52116,0.52342,0.36627,
    	0.37058,0.53488,0.52774,0.36507,
    	0.27002,0.37003,0.36238,0.26085,
    
    	0.4002,0.54017,0.55382,0.42993,
    	0.61098,0.88016,0.88922,0.63957,
    	0.64538,0.92468,0.91874,0.65217,
    	0.49472,0.67493,0.66578,0.49065
    
    Layer 3 activation:
    	0.52116,0.52342,
    	0.53488,0.52774,
    	
    	0.88016,0.88922,
    	0.92468,0.91874
    
    target:
    	0.7,0.6,
    	0.5,0.4,
    
    	0.3,0.2,
    	0.1,0
    
    Layer 3 blame:
    	0.17884,0.07658,
    	-0.03488,-0.12774,
    
    	-0.58016,-0.68922,
    	-0.82468,-0.91874
    
    Layer 2 blame:
    	0,0,0,0,
    	0,0.17884,0.07658,0,
    	0,-0.03488,-0.12774,0,
    	0,0,0,0,
    
    	0,0,0,0,
    	0,-0.58016,-0.68922,0,
    	0,-0.82468,-0.91874,0,
    	0,0,0,0
    
    Layer 1 blame: 
    	0,0,0,0,
    	0,0.17884,0.07658,0,
    	0,-0.03488,-0.12774,0,
    	0,0,0,0,
    
    	0,0,0,0,
    	0,-0.58016,-0.68922,0,
    	0,-0.82468,-0.91874,0,
    	0,0,0,0
    
    Layer 0 blame: 
    	-0.1021612,-0.2424868,-0.2526264,-0.1485652,
    	-0.2912204,-0.6655076,-0.6947076,-0.3948608,
    	-0.3290468,-0.7531076,-0.7823076,-0.4446344,
    	-0.2285932,-0.5069644,-0.5260248,-0.2907052
    
    gradient: 
    	-6.65352,
    	-2.27731944,-3.09769472,-2.82596468,
    	-4.35582312,-5.5801104,-4.87145808,
    	-4.40816792,-5.51474528,-4.72076836,
    	
    	0.0928,
    	-0.02012312,-0.01653216,0.00021995999999997,
    	-0.01459476,-0.0034554,0.01851276,
    	0.05737384,0.08298856,0.08954992,
    	
    	-3.0128,
    	-0.58561972,-0.77292296,-0.68036924,
    	-1.03960116,-1.2990522,-1.09834164,
    	-0.90605436,-1.10450424,-0.91155168
    
    updated weights: 
    	-0.0665352,
    	-0.0127731944,-0.0109769472,0.0017403532,
    	-0.0035582312,-0.005801104,0.0112854192,
    	0.0259183208,0.0248525472,0.0427923164,
    
    	0.100928,
    	0.1097987688,0.1198346784,0.1300021996,
    	0.1398540524,0.149965446,0.1601851276,
    	0.1705737384,0.1808298856,0.1908954992,
    
    	0.169872,
    	0.2041438028,0.2122707704,0.2231963076,
    	0.2296039884,0.237009478,0.2490165836,
    	0.2609394564,0.2689549576,0.2808844832