Time series prediction
In this project, you will regress a neural network with sinusoidal activation functions to fit to some timeseries data.
Instructions:
 Download this data.
(I scraped this data from the US department of labor statistics.)
 Create a neural network with one hidden layer of 101 neurons.
This network will accept one continuous input value (time), and one continous output value (the unemployment rate indicated in the data).
Use the sinusoid activation function in the first 100 units of the hidden layer.
Use the identity activation function for the last unit of the hidden layer.
Initialize the weights in the hidden layer as follows:
weight bias
 
unit 1 1*2*pi pi
unit 2 2*2*pi pi
unit 3 3*2*pi pi
... ... ...
unit 50 50*2*pi pi
unit 51 1*2*pi pi/2
unit 52 2*2*pi pi/2
unit 53 3*2*pi pi/2
... ... ...
unit 100 50*2*pi pi/2
unit 101 0.01 0 (the linear unit)
Use the identity activation function in the output layer.
Initialize the weights in the output layer with small random values.
 Train this network using the feature values
0/256
1/256
2/256
...
255/256
and the first 256 rows from the labor statistics data as the corresponding labels.
 Plot the data and corresponding predictions for values from 0/256 to 356/256.
Label the axes.
Indicate the point t=256/256 on your chart, pehaps with a vertical line.
(This is the point where it begins predicting into the future, as far as it knows.)
 Implement L^{1} regularization.
Regularize only the output layer during training.
Fiddle with the regularization term until you find one that makes a difference, but doesn't ruin the results.
Add these results to the same chart.
 Implement L^{2} regularization.
Regularize only the output layer.
Fiddle with the regularization term until you find one that makes a difference, but doesn't ruin the results.
Add these results to the same chart.
Now, you should have four curves/lines (the data, predictions made without regularization, predictions with L1 regularization, and predictions with L2 regularization).
Label them, so it is clear which is which.
 Submit an archive containing your code and chart here.
There is no need to execute anything on the submission server.
Hints:
 Here is some debug spew from a working implementation to help you debug a broken implementation.
To use it, set all the bias values to 0 and all the weights in the output layer to 0.01.
Also, present the training patterns in sequential order.
