Re: Neural networks from scratch in forth

Hi again,
A last example concerns digit recognition.
The digit is in {0, 1, ...,9}.
The sapmles (examples) are 8x8-matrices (patterns for digits).
The example begins here, it consists of three files:
    - numbers_data.fs
    - use_neural_net.fs
    - net_predict.fs
These files are given below:
\ ------------numbers_data.fs---------------------
\ data samples numbers
variable n_samples1
10 n_samples1 !
: _  0e f, ;
: X  1e f, ;
create data1
\ sample 1
\ inputs
_ _ _ _ _ _ _ _
_ _ X X X _ _ _
_ X _ _ _ X _ _
_ X _ _ _ X _ _
_ X _ _ _ X _ _
_ X _ _ _ X _ _
_ _ X X X _ _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  X  _  _  _  _  _  _  _  _  _
\ sample 2
\ inputs
_ _ _ _ _ _ _ _
_ _ _ X _ _ _ _
_ _ X X _ _ _ _
_ X _ X _ _ _ _
_ _ _ X _ _ _ _
_ _ _ X _ _ _ _
_ _ X X X _ _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  _  X  _  _  _  _  _  _  _  _
\ sample 3
\ inputs
_ _ _ _ _ _ _ _
_ _ X X X _ _ _
_ X _ _ _ X _ _
_ _ _ _ X _ _ _
_ _ _ X _ _ _ _
_ _ X _ _ _ _ _
_ X X X X X _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  _  _  X  _  _  _  _  _  _  _
\ sample 4
\ inputs
_ _ _ _ _ _ _ _
_ _ X X X _ _ _
_ X _ _ _ X _ _
_ _ _ X X _ _ _
_ _ _ _ _ X _ _
_ X _ _ _ X _ _
_ _ X X X _ _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  _  _  _  X  _  _  _  _  _  _
\ sample 5
\ inputs
_ _ _ _ _ _ _ _
_ _ _ _ X _ _ _
_ _ _ X X _ _ _
_ _ X _ X _ _ _
_ X _ _ X _ _ _
_ X X X X X _ _
_ _ _ _ X _ _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  _  _  _  _  X  _  _  _  _  _
\ sample 6
\ inputs
_ _ _ _ _ _ _ _
_ X X X X X _ _
_ X _ _ _ _ _ _
_ X X X X _ _ _
_ _ _ _ _ X _ _
_ X _ _ _ X _ _
_ _ X X X _ _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  _  _  _  _  _  X  _  _  _  _
\ sample 7
\ inputs
_ _ _ _ _ _ _ _
_ _ X X X _ _ _
_ X _ _ _ _ _ _
_ X X X X _ _ _
_ X _ _ _ X _ _
_ X _ _ _ X _ _
_ _ X X X _ _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  _  _  _  _  _  _  X  _  _  _
\ sample 8
\ inputs
_ _ _ _ _ _ _ _
_ X X X X X _ _
_ X _ _ _ X _ _
_ _ _ _ X _ _ _
_ _ _ X _ _ _ _
_ _ _ X _ _ _ _
_ _ _ X _ _ _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  _  _  _  _  _  _  _  X  _  _
\ sample 9
\ inputs
_ _ _ _ _ _ _ _
_ _ X X X _ _ _
_ X _ _ _ X _ _
_ _ X X X _ _ _
_ X _ _ _ X _ _
_ X _ _ _ X _ _
_ _ X X X _ _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  _  _  _  _  _  _  _  _  X  _
\ sample 10
\ inputs
_ _ _ _ _ _ _ _
_ _ X X X _ _ _
_ X _ _ _ X _ _
_ _ X X X X _ _
_ _ _ _ _ X _ _
_ X _ _ _ X _ _
_ _ X X X _ _ _
_ _ _ _ _ _ _ _
\ outputs
\ 0  1  2  3  4  5  6  7  8  9
  _  _  _  _  _  _  _  _  _  X
: _ 0e ;
: X 1e ;
\ -----------------numbers_data.fs ends here---------------------
\ -----------------use_neural_net.fs----------------------
include neural_networks.fs
\ including data
include numbers_data.fs
data1 >data
n_samples1 @  >n_samples
\ neuralnet: 64 inputs, 10 outputs and 1 hidden layer (5 neurons each)
10 10 64 1 neuralnet: net1
' net1 is net
net_layers
' dlsigmoid is act_func
' dlsigmoid is act_func_ol
1e-3 >eta
9e-1 >beta
1000000 >epochs
1e-2 >tol
100 >display_step
false >adapt_eta
true >init_net
learn
\ --------------use_neural_net.fs ends here-----------------
\ --------------net_predict.fs---------------------------
include use_neural_net.fs
: .result
    0
    0e
    n_outputs @ 0 do
      outputs i floats + f@
      fover fover f< if
        drop
        fnip
        i
      else
        fdrop
      then
    loop
    cr
    cr s" The result:" type
    cr s" -----------" type
    cr s" It is: " type . s" with probability: " type f.
    cr
;
\ Learning
learn
\ predictions
cr
cr s" Now, prediction:" type
cr s" ----------------" type
\ change the inputs and see if the neuralnet recognizes it
\ the inputs
: X 1e ;
: _ 0e ;
cr
cr s" First example:" type
cr s" --------------" type
cr
_ _ _ _ _ _ _ _
_ _ X X X _ _ _
_ X _ _ _ X _ _
_ _ _ X _ _ _ _
_ _ _ _ X _ _ _
_ X _ _ _ X _ _
_ X X X X _ _ _
_ _ _ _ _ _ _ _
to_inputs
forward_pass outputs_probs   .outputs .result
cr
cr
cr s" Second example:" type
cr s" ---------------" type
cr
_ _ _ _ _ _ _ _
_ _ X X X _ _ _
_ X _ _ _ X _ _
_ _ _ _ X _ _ _
_ _ _ X _ _ _ _
_ _ X _ _ _ _ _
_ _ X X X _ _ _
_ _ _ _ _ _ _ _
to_inputs
forward_pass outputs_probs   .outputs .result
cr
cr
cr s" Third example:" type
cr s" -------------" type
cr
_ _ _ _ _ _ _ _
_ _ _ _ X _ _ _
_ _ _ X _ _ _ _
_ _ _ X _ _ _ _
_ _ _ X _ _ _ _
_ _ _ X _ _ _ _
_ _ X _ _ _ _ _
_ _ _ _ _ _ _ _
to_inputs
forward_pass outputs_probs   .outputs .result
cr
cr
cr s" Fourth example:" type
cr s" ---------------" type
cr
_ _ _ _ _ _ _ _
_ _ _ _ X _ _ _
_ _ _ X _ X _ _
_ _ X _ _ X _ _
_ X X X X X _ _
_ _ _ _ _ X _ _
_ _ _ _ _ X _ _
_ _ _ _ _ _ _ _
to_inputs
forward_pass outputs_probs   .outputs .result
cr
cr
cr s" fifth example:" type
cr s" ---------------" type
cr
_ _ _ _ _ _ _ _
_ X X X X X _ _
_ _ _ _ _ X _ _
_ _ _ _ _ X _ _
_ _ _ _ _ X _ _
_ _ _ _ _ X _ _
_ _ _ _ _ _ _ _
_ _ _ _ _ _ _ _
to_inputs
forward_pass outputs_probs   .outputs .result
cr
\ --------------net_predict.fs ends here----------------
In the terminal (command line), type:
gforth net_predict.fs
or include it in gforth, iForth or VfxForth.
\ Prediction: possible forms
\ net_predict
\ net_predict_probs
\ net_predict_softmax
\ net_predict_ips
\ forward_pass .outputs .result
\ forward_pass outputs_ident   .outputs .result
\ forward_pass outputs_probs   .outputs .result
\ forward_pass outputs_softmax .outputs .result
\ forward_pass .result
\ forward_pass outputs_ident   .result
\ forward_pass outputs_probs   .result
\ forward_pass outputs_softmax .result
Enjoy.
Ahmed
--