Leaning toward the wall for balance is a common problem with green horses that don't trust their own balance. Instead of accepting guidance and framing from the rider's legs and seat, the horse "pulls" toward the wall for support, like a balancing beam.
To combat this, I first recommend spending most of your ride on the second track and quarter line, so your horse becomes more comfortable being off the wall and in turn, relaxes under you. Riding off the rail also gives the rider a better feeling for the horse's actual straightness. Start this work at the walk and only trot if your horse feels comfortable off the wall and has relaxed.
Along with walking off the wall, you can also include walk-halt transitions. In the transitions, I make sure to differentiate between a passive, or holding leg and seat, and a driving leg and seat. I use mostly leg (closing of the knees and thighs into the thigh block or into the saddle), seat and a tiny bit of rein-closing or wiggling my fingers. If your horse doesn't listen to your leg and seat aids in the downward transition--doesn't halt--use a stronger rein aid as a correction, then give instantly with your hand and leg. In the upward transitions I use my lower leg. This can only work if the rider isn't gripping with her leg the whole time. If you are always gripping, the horse won't be able to tell when you actually want a reaction from your leg. As your horse steps on in the upward transition, try to be light on his back. Let him take your upper body forward.
When everything is going well, proceed into trot-walk transitions. Do only short sections of trot--maybe halfway down the long side--then ride a quiet transition to walk. Trot on again, and after another short section of trot do another downward transition and so on. Use your legs in the transitions as described earlier. If your horse's back is tight, go into a rising trot, but sit for the downward transitions. This will enable better use of your knees and thighs and make it easier to differentiate between your leg aids.
As your horse becomes comfortable with this exercise, he'll relax--his rhythm and gait will loosen and his stride will be a bit bigger. As he moves more over his back, he'll start to swing in the trot. At the walk, he should quietly march with a good energy level and in a decent balance without leaning.
Once the horse is quiet, has accepted being off the rail and has started to relax, proceed with turns on the forehand and leg yields. To do a turn on the forehand to the left, gently ask your horse to step sideways around his forehand using a sideways driving right leg, slightly behind the girth. Increase your weight a bit on the left, because I want to teach the horse to step under my weight and in the direction of the movement, like in canter pirouette. When you drive with the right leg, you must receive with the left leg, which is also slightly behind the girth. Your horse doesn't need to be perfectly on the bit, but she should be reasonably accepting the contact.
When you put your leg on to ask your horse to take a sideways step behind, his first reaction might be to plow forward. This reaction can make the horse hollow, which ultimately results in the rider having to pull back on the reins. Ideally, you should have a little bit of neck control so the horse is able to be round and on the bit. If done in a quiet way, this exercise can actually help put the horse on the bit. When doing a turn on the forehand to the left, the horse moves off the rider's right leg, which connects him into the left rein and left leg.
Train the turn on the forehand away from the wall. Halt parallel to the wall, but with about a horse-length distance to the wall. You don't want the horse to get claustrophobic when he turns his head toward the wall.
I begin a turn on the forehand by doing one or two steps of it, then halting, putting the horse back on the bit if he has come off and then letting him relax for a moment.
You can also leg yield along the wall and then away from it. The aids for leg yield are similar to those for the turn on the forehand, except that the horse is moving sideways with both his forehand and hindquarters. For a green horse, accept a green leg yield. It doesn't have to be a leg yield from point A to point B. What is important is that your horse starts to understand to move away from one leg and accept the other one as a guide, as he learns to stay between your legs.
A natural reaction for a green horse is to run through the leg when asked to move away from it. I correct this by walking straight, leg yielding for three or four steps, then going straight ahead again. When the horse starts to fall or fling sideways he needs to go straight again to rebalance and realign.
When a horse runs from the leg, the rider often takes his legs away from the horse's sides. This is the opposite of what you should do. If the horse overreacts to the leg, keep your leg softly on your horse's barrel. Relax the horse and repeat, giving a light aid. Also, make sure that the leg aid doesn't surprise your horse. Teach your horse to accept and move away from the leg in a quiet and understanding way. I encourage riders to have soft, breathing calves at all times, just touching the horse's ribcage and barrel. Then the rider can increase the pressure, squeeze for a moment to create an aid and then relax the leg again. The turn on the forehand and the leg yield should make the horse less tense, less crooked and able to accept both of the rider's legs.
If the horse doesn't react to the leg aid, that's when a correction such as a tiny kick or a little touch with the whip is needed. After that correction, retest the acceptance of your aid. Ask yourself: Has my horse understood it? Give a smaller, quieter aid again. Remember, we are training horses to react to small aids, not just to react off a correction.
Other helpful exercises include riding serpentines, 10-meter half circles and figure eights when changing direction. This will allow the horse to gradually spend more time in the middle of the ring and not rely on the wall.
A big mistake often made is to use too much inside rein, or bending in the neck in an attempt to lead the horse off the wall. It might look like the horse is straight and off the wall, but in actuality this allows the horse to fall through his outside shoulder and be even more stuck on the wall.
Be very aware of your seat structure and leg position. Often when we have problems like overcompensating by bending, our position as a rider is negatively affected. We become crooked and it makes the horse fall more sideways and crooked.
When working on solving this problem, take your time and keep checking your seat structure and leg aids. If it's been happening for a long time, it is going to take a while for you to retrain your horse and for your horse to feel confident and comfortable being off the wall.