Our last weekend in New Caledonia, we indeed went horseback riding - at Yala Ranch in Dumbea, just outside Noumea.
I had been up on a horse once or twice at school when I was younger and had been traumatised (as I am wont to do by anything bigger or faster than myself). Laurent had ridden a few times before.
Yala Ranch turned out to be the perfect place to go. We spent a couple of hours on the horses, riding through the river, climbing a gentle hill, getting a lay of the land. The guides were very friendly and reassuring - and though Laurent was nearly thrown (he loves to tease the horses), I managed to stay on my horse.
There are several options for horseriding in New Caledonia. I would encourage you to give it a try - for those days when it feels a little too cool (under 20 degrees) to go scuba diving.