Alright so this is a somewhat late response because I had to reread a few things to fully understand, but I did read everything multiple times. First off just to let you know you did not come off bad in anyway with this post. I’m glad I got someone enthusiast about this to respond and help me. Another thing is I actually have done a lot of the ideas you gave beforehand, maybe in different ways though. I am trying this with tensorflow, but I have made this code before using a java based neural network library that I got while following the toy NN coding train videos.
The way I trained the java based NN mouse tracker is by having a normal ball that follows a point that is changed to a random position after a set amount of time using the steer algorithm(which i got from coding train video, with a title somewhat about steering agents). The normal ball is easily able to go to the target point and has no ml involved with it. Then I have a set amount of NN balls that have a similar algorithm as the normal ball except instead of using the steering algorithm to find the acceleration applied I use the NN feedforward to find the acceleration I should apply. I train each NN by passing in the x and y position of the NN ball and making the expected output the acceleration that the normal ball has at each frame. This way I have a best output that I can say that the NN should be giving each frame. Any NN ball that does not reach the target point by time the target resets is deleted and when there is one NN ball left I save its NN and generate a new set of NN balls that have a mutation of the saved NN. So I think I actually had very similar ideas to you. The only problem is I thought this program was working since the NN balls were able to slowly get better at solving the problem, but after looking at my code I saw that a lot of stuff was done wrong and now i’m actually going back and trying to fix the java based code now before I do the tensorflow one. Sorry for the switch I hope you can still help though since it is the same problem just different code language.
Some things that I have changed with the code based on your help is now instead of just the x and y pos inputs I have the x pos, y pos, x vel, y vel, x tar, and y tar as the inputs. I still generate a random position after a set time to be followed in order to train. The problem I get now is that the output from the feedforward is slowly getting smaller and the two output values (x acc and y acc) are always positive so the NN ball is always moving towards the bottom right and a slowing rate.
The blue ball is the steering algorithm ball, the green is the NN ball, and the red dot is the target.
I made the testing to just one NN ball for now because I want to make sure that I have the training part is correct, however based on what I said before and the video it seems like it is not. I do plan to have the GA part implemented later, but I want the NN to be able to do some learning then the best is taken when all NN balls get deleted.
Main code:
int size = 1;
int rows = 2;
int cols = 1;
int count = 0;
int limit = 150;
ArrayList<Ball> ball;
ArrayList<NN> n;
Ball2 ball2;
Matrix tar;
Matrix hold;
Matrix hold2;
void setup(){
size(500,500);
ball = new ArrayList<Ball>();
ball2 = new Ball2();
tar = new Matrix(2,1);
n = new ArrayList<NN>();
hold = new Matrix(6,cols);
hold2 = new Matrix(rows,cols);
for(int i = 0; i < size; i++){
ball.add(new Ball());
n.add(new NN(6,8,2));
}
}
void draw(){
background(0);
//Generates new target if the counter reaches max time
if(count == limit){
tar.data[0][0] = (random(width));
tar.data[1][0] = (random(height));
count = 0;
NN fake = n.get(0);
fake.printNN();
n.remove(0);
ball.remove(0);
ball.add(new Ball());
n.add(fake);
}
//When 1 ball is left create new balls mutated from the last one
//if(ball.size() == 1){
// NN fake = n.get(0);
// fake.printNN();
// n.remove(0);
// ball.remove(0);
// println(n.size());
// println(ball.size());
// for(int i = 0; i < size; i++){
// ball.add(new Ball());
// n.add(fake.mutate());
// }
// limit = 100;
// count = 0;
//}
//loop through each NN ball updating feedingforward and updating position
for(int i = 0; i < ball.size(); i++){
//make input matrix
hold.data[0][0] = ball.get(i).pos.x;
hold.data[1][0] = ball.get(i).pos.y;
hold.data[2][0] = ball.get(i).vel.x;
hold.data[3][0] = ball.get(i).vel.y;
hold.data[4][0] = tar.data[0][0];
hold.data[5][0] = tar.data[1][0];
//make expected output matrix of NN
hold2.data[0][0] = ball2.acc.x/ball2.maxForce;
hold2.data[1][0] = ball2.acc.y/ball2.maxForce;
//train the nn with hold and hold2
n.get(i).train(hold,hold2);
//pass through ball's respective nn's prediction as acceleration
println(" x " +n.get(i).feedforward(hold).data[0][0]);
println(" y " +n.get(i).feedforward(hold).data[1][0]);
ball.get(i).update((n.get(i).feedforward(hold).multiply(ball.get(i).maxForce)));
ball.get(i).show();
//n.get(i).printNN();
}
//update and show steering algorithm ball
ball2.update(new PVector((float)tar.data[0][0],(float)tar.data[1][0]));
ball2.show();
fill(150,50,50);
ellipse((float)tar.data[0][0],(float)tar.data[1][0],10,10);
//delay(200000);
count++;
}
NN ball code:
class Ball{
PVector pos, vel, acc, accT;
float maxSpeed = 1, maxForce = 5;
Matrix accOut = new Matrix(2,1);
boolean tarHit = false;
Ball(){
pos = new PVector(random(width),random(height));
vel = new PVector(1,1);
acc = new PVector(0,0);
}
void update(Matrix accel){
acc = new PVector((float)accel.data[0][0],(float)accel.data[1][0]);
//println("acc x : "+acc.x + "acc y : " +acc.y);
vel.add(acc);
pos.add(vel);
//acc.mult(0);
if(this.pos.x > tar.data[0][0]-5 && this.pos.x < tar.data[0][0]+5 &&
this.pos.y > tar.data[1][0]-5 && this.pos.y < tar.data[1][0]+5){
this.tarHit = true;
}
//boundaries();
}
void show(){
fill(150,250,150);
ellipse(pos.x,pos.y,30,30);
}
void boundaries(){
if(pos.x > width) pos.x = 0;
if(pos.x < 0) pos.x = width;
if(pos.y > height) pos.y = 0;
if(pos.y < 0) pos.y = height;
}
}
I should let you know that as I am typing this reply I am also changing the code so my train of thought might seem off. I have tested your idea of just changing the velocity instead of applying the acceleration so now I have the steering algorithm ball’s velocity x and y as the expected prediction for training and the same 6 inputs as before. Also the update function for the NN ball has been changed to the following
void update(Matrix accel){
acc = new PVector((float)accel.data[0][0],(float)accel.data[1][0]);
vel = acc;
pos.add(vel);
//acc.mult(0)
if(this.pos.x > tar.data[0][0]-5 && this.pos.x < tar.data[0][0]+5 &&
this.pos.y > tar.data[1][0]-5 && this.pos.y < tar.data[1][0]+5){
this.tarHit = true;
}
//boundaries();
}
Now the velocity is changed directly, however I just noticed that when I am training the NNs it makes no sense to put the acc or vel of the steering algorithm ball as the expected output because that ball is at a different pos so I need to simulate a steering algorithm at the position of each NN ball(which I also now see if probly what you meant by optimal velocity). If I do this it would be hard to just change the velocity since the velocity at that pos can be multiple values, but the steering(acceleration) at a position is always the same no matter the velocity, so I am going to keep changing the acceleration. I tested this out and I got really good results!
(video in below post)
This is without the GA part still. I just set the NN balls back to pos 0,0 when the target point is changed and let them keep their NN. I plan on adding the GA however I think it might need some more thought to the fitness. If I tried to just take how well it gets close to the target it might make it so i get a bad NN that was lucky enough to spawn close to the target, but I will try it out and see how it goes. I know I pretty much changed the topic of this post, but I will put the tensorflow part on pause for now then continue after I implement the GA part for the java mouse follow code so that I have a good template of how it should be structured.
I hope i wasn’t too confusing and I hope you don’t mind me pretty much typing out my thoughts/process instead of just saying my ends results. I think it helps to show how I got to where I am now and what I have tried, also i’m a little too lazy to revise the above stuff. I think I followed most of the advice you gave hopefully I didn’t miss any important things, there was quite a bit though.
As for my experience with this stuff I would say I have very little. Most of what I know is from coding train and some of that andrew ng course(i didn’t finish it). With tensorflow all I know is from coding train videos and with javascript I am very new to that where I don’t fully understand how the language is structured but it is similar to java so I can somewhat get by.