Neural Networks & SVMs

[BHURRICANE]

Would anyone care to share any success or failures implimenting neural networks, support vector machines, kernel machines, etc. to assist in the handicapping process? Specifically, I have been exploring various encoding and reduction schemes to the race data while at the same time preserving the intrinsic representations in lower dimensions. Any advice or experiences would be helpful. Thanks.

[hurrikane]

Hey,
what's up with the name dude?

[hurrikane]

I am no NN guru but my experience is that there is a lot of noise and redundancy in the data. Not great results. do you intend on using drf data?

But I am still working on it.

[ceejay]

You might try googling "neural network horse." There was a guy that used to hang out here who had a NN program for sale, but I haven't seen him for a while....

[DJofSD]

I started a project many years ago involving NN. I never got very far. Beyond the initial purchase of a few books and workbooks, I could not figure out if the various pre packaged NN engines would work for me. And they're too expensive to try too many of them doing a trail and error.

This was many years ago. I'm sure things have changed in the meantime.

I'd be interested in learning anything more about NN and handicapping.

DJofSD

[Grifter]

OK, I'm impressed. I had to look up SVM.... read all about it. Still haven't the foggiest....

I also fooled around with NN. Frankly, it seems too much "black-box" for my tastes. I cannot get over the fear of "over-fitting".

I think I trust multiple regression analysis, in all its modes, to NN. (I'd lump SVM with NN as well, if I thought I knew what I was talking about.)

-- Grifter

[kenwoodallpromos]

I want to know how he got a "dstinguished" remark after 1 post!

[ceejay]

I think I trust multiple regression analysis, in all its modes, to NN.In my experience (not racing) MLA regression works as well as NN in most cases: I have the same fears as you about over-fitting. I have had some (but rare) situations where "fuzzy logic" (which is similar to NN) has worked but I view it, and NN as a "last resort."

Fuzzy logic links:
http://www.oilfield.slb.com/media/services/software/geo/int_petro_advmod_techsheet.pdf
http://www-2.cs.cmu.edu/Groups/AI/html/faqs/ai/fuzzy/part1/faq-doc-2.html

[Dave Schwartz]

BHurrikane,

Not sure what you are looking for here but I can tell you my experience with neural net technology has both strong points and weak points. As for vector machines, this is a new approach and I doubt there will be much if anything out there as it applies to horse racing.

Neural nets are good at making "generalizations," and, as such, are theoretically a good match for horse races. The problem is that they have a built in sensitivity to "bad data." To that end, "bad data" (however you choose to define it) must be culled from the data set. IMHO, this makes neural nets a poor choice for horse racing.

A better choice is the genetic algorithm approach.


Where are you in this process? Are you looking to develop something for yourself or purchase something already developed? Are you a software engineer or a hobbiest?

There are some very astute people here and if you give them a better question, I think you will receive some meaningful answers.

Hmm... The above package makes me ask, "Where is GameTheory these days?"


Regards,
Dave Schwartz

[BHURRICANE]

I am no NN guru but my experience is that there is a lot of noise and redundancy in the data. Not great results. do you intend on using drf data?

But I am still working on it.

This is exactly why I'm interested in using a neural network. In the little exploring I've done on the internet, I've found that the neural network implimentations to date are very unsophisicated with their approach in "presenting" the data to the neural network. There are almost no provisions for for standardization of the data as well as the removal of "outlier" or errand data that would effect the performance. I think I have those situations accounted for.

[BHURRICANE]

OK, I'm impressed. I had to look up SVM.... read all about it. Still haven't the foggiest....

I also fooled around with NN. Frankly, it seems too much "black-box" for my tastes. I cannot get over the fear of "over-fitting".

I think I trust multiple regression analysis, in all its modes, to NN. (I'd lump SVM with NN as well, if I thought I knew what I was talking about.)

-- Grifter

Over-fitting can be a problem. This can be avoided with a properly designed neural network however. If the learning rate and momentum is adjusted in concert with monitoring the rate at which the absolute error is decending, you can avoid over-fitting (or over-training) a network.

Multiple regression analysis is useful for a single horse evaluation, however, seems to lose some utility when applied to a race as a whole for me. I'm interested in pattern identification as well as the classification aspects that a neural network can provide.

[BHURRICANE]

BHurrikane,

Not sure what you are looking for here but I can tell you my experience with neural net technology has both strong points and weak points. As for vector machines, this is a new approach and I doubt there will be much if anything out there as it applies to horse racing.

Neural nets are good at making "generalizations," and, as such, are theoretically a good match for horse races. The problem is that they have a built in sensitivity to "bad data." To that end, "bad data" (however you choose to define it) must be culled from the data set. IMHO, this makes neural nets a poor choice for horse racing.

A better choice is the genetic algorithm approach.


Where are you in this process? Are you looking to develop something for yourself or purchase something already developed? Are you a software engineer or a hobbiest?

There are some very astute people here and if you give them a better question, I think you will receive some meaningful answers.

Hmm... The above package makes me ask, "Where is GameTheory these days?"


Regards,
Dave Schwartz

I have been using a hybrid GA / Neural Net and had large sucess with it.

http://www.cs.utexas.edu/users/kstanley/neat.html

The NEAT model allows for the idea of "competition" among entities with
the GA approach and the neural side is used as a learning utility. I think
this is fantastic work that might revitalize some interest in neural networks
as a tool to facilitate a "learning" system. Simple back-prop isn't enough alone.

Your correct in that the trash data must be removed. Right now my software
tosses the "exceptional" data values away.

Also, I have been using a little ID3, for decision tree style approaches to
consider what data to put into the system. This acts alot like a person
does to pre-filter the data under consideration.

I am a software engineer by trade, and I have a hobbiest interest in horseracing.

[BHURRICANE]

OK, I'm impressed. I had to look up SVM.... read all about it. Still haven't the foggiest....

I also fooled around with NN. Frankly, it seems too much "black-box" for my tastes. I cannot get over the fear of "over-fitting".

I think I trust multiple regression analysis, in all its modes, to NN. (I'd lump SVM with NN as well, if I thought I knew what I was talking about.)

-- Grifter

I do want to comment here that SVM's are much more effective in their output as it relates to horse racing than neural networks in my experiments so far. They aren't subject to some of the tainting that can occur with back-prop networks. Hybrid approaches to kernel style computing is proving quite an effective means to achieve more human quality decision making. SVM is a great approach to integrating regression based systems.

[Dave Schwartz]

BH,

I have been using a hybrid GA / Neural Net and had large sucess with it.

http://www.cs.utexas.edu/users/kstanley/neat.html

The NEAT model allows for the idea of "competition" among entities with
the GA approach and the neural side is used as a learning utility. I think
this is fantastic work that might revitalize some interest in neural networks
as a tool to facilitate a "learning" system. Simple back-prop isn't enough alone.

Your correct in that the trash data must be removed. Right now my software
tosses the "exceptional" data values away.

Most interesting. My personal approach is towards genetic "agents" that are (somewhat) free to roam the database building rules and assigning points as they see fit.

When I built my first "competitor" neural net in ThoroBrain2, c1990 (a couple of years before the BrainMaker guy went public with it), I found that it was highly predictive. But there was always the problem of culling the bad races.

And the tough part of that is that not all bad races are obviously so. I mean, the race where the 4 horse falls down and takes the 3 inside horses with him is obvious.

The problem is the 4/5, wire-to-wire monster who never comes out of the gate. Is that an abberation or is it predictable within the data?

If it is not predictable within the data, then the race should be removed.

The problem is that this is a pretty regular occurence, happening several times per card. (Maybe not with this precise example, but I am sure you get the idea.)

The important part of your quote is:

I have been using a hybrid GA / Neural Net and had large sucess with it.

That says a lot.

I wish you continued success.


Regards,
Dave Schwartz

[BHURRICANE]

BH,



Most interesting. My personal approach is towards genetic "agents" that are (somewhat) free to roam the database building rules and assigning points as they see fit.

When I built my first "competitor" neural net in ThoroBrain2, c1990 (a couple of years before the BrainMaker guy went public with it), I found that it was highly predictive. But there was always the problem of culling the bad races.

And the tough part of that is that not all bad races are obviously so. I mean, the race where the 4 horse falls down and takes the 3 inside horses with him is obvious.

The problem is the 4/5, wire-to-wire monster who never comes out of the gate. Is that an abberation or is it predictable within the data?

If it is not predictable within the data, then the race should be removed.

The problem is that this is a pretty regular occurence, happening several times per card. (Maybe not with this precise example, but I am sure you get the idea.)

The important part of your quote is:



That says a lot.

I wish you continued success.


Regards,
Dave Schwartz


I agree. I remove any race that doesn't "feel" right. We cannot plan or predict strange occurances. I simply omit the race and move on. I'm focused on how the race would run under ideal circumstances and conditions, and then hope that the race resembles those to a certain degree, and subsequently the results are somewhat more predictable in addition to assistance in identifying overlays etc. Systems are programmed and based on rules not exceptions, humans are exception handlers. If I see something I think my program "missed" then I back test it to determine that or find out if I'm seeing something that doesn't "normally" occur.

[BHURRICANE]

BH,



Most interesting. My personal approach is towards genetic "agents" that are (somewhat) free to roam the database building rules and assigning points as they see fit.

When I built my first "competitor" neural net in ThoroBrain2, c1990 (a couple of years before the BrainMaker guy went public with it), I found that it was highly predictive. But there was always the problem of culling the bad races.

And the tough part of that is that not all bad races are obviously so. I mean, the race where the 4 horse falls down and takes the 3 inside horses with him is obvious.

The problem is the 4/5, wire-to-wire monster who never comes out of the gate. Is that an abberation or is it predictable within the data?

If it is not predictable within the data, then the race should be removed.

The problem is that this is a pretty regular occurence, happening several times per card. (Maybe not with this precise example, but I am sure you get the idea.)

The important part of your quote is:



That says a lot.

I wish you continued success.


Regards,
Dave Schwartz

On that same thought, I like to reinforce learning behavior that is expected.
I've always been interested in things like resturants that advertise "low" volume times offering discounts, etc. Why not advertise what already is doing well even more and increase profits? I see horse racing the same way, I want to boost and reinforce what does well, not focus on what I missed. This also can greatly reduce the complexity of a system, such that, I mark up a blunder to "my system doesn't understand that" and avoid trying too, and hone the angles it does very well at, then exclusively play those races.

[hurrikane]

Curious,

with over 300000 races run every year how are you removing races that don't 'feel right' ?

[hurrikane]

sorry, that's should be 50,000 races

[osophy_junkie]

I have been successfully using a genetic algorithem to create SVM models for about 4 months. Throwing row data in a SVM did little good. Normalization, segragation (maybe called stacking and/or boosting), and cleaning up the data were necessary. I use libsvm[1] and a customized version of scipy.ga[2].

When I say cleaning up data, I mean hand timing races, recording various features of the trip, spelling corrections to Jockeys/Owners. I do not believe in removing races that do not "feal right". A system without the noise is an over simplication of the problem. This may be useful when solving physics problems, but does not allow your system to develop the maturatity required for long term stability.

The attributes used in the NN article were specific to dog racing and if they ever did work for horse racing do not work now. Papers by Benter and Bolton & Chapman discuss what makes for good attributes. The normalization I started off with mimics the multinomial logit they each use.

Ed

[1] http://www.csie.ntu.edu.tw/~cjlin/libsvm/
[2] http://www.scipy.org/

[BHURRICANE]

Curious,

with over 300000 races run every year how are you removing races that don't 'feel right' ?

I only use about 2,000 total races in my system. So I review them manually.

[ratpack]

Not sure if this is of any help.

Bris has that free Neurax and Neurax Pro software on their Web site, that is suppose to be neural network.

[BHURRICANE]

I have been successfully using a genetic algorithem to create SVM models for about 4 months. Throwing row data in a SVM did little good. Normalization, segragation (maybe called stacking and/or boosting), and cleaning up the data were necessary. I use libsvm[1] and a customized version of scipy.ga[2].

When I say cleaning up data, I mean hand timing races, recording various features of the trip, spelling corrections to Jockeys/Owners. I do not believe in removing races that do not "feal right". A system without the noise is an over simplication of the problem. This may be useful when solving physics problems, but does not allow your system to develop the maturatity required for long term stability.

The attributes used in the NN article were specific to dog racing and if they ever did work for horse racing do not work now. Papers by Benter and Bolton & Chapman discuss what makes for good attributes. The normalization I started off with mimics the multinomial logit they each use.

Ed

[1] http://www.csie.ntu.edu.tw/~cjlin/libsvm/
[2] http://www.scipy.org/

Noise removal for me means to remove ridiculous or errant data found in PP files. In addition I consider races that could not be handicapped by a human to be noise itself. These removals have made my system more effective, and perhaps with some experiments, you might also find the same in yours.

On the GA note, it's been very useful in feature (data input) selection(s).
In that it has allowed reduction of input data for processing while retaining it's effectiveness.

Generally speaking are you approaching horse racing as a regression or classification problem or both? How have you divided your datasets? It seems that certain horse racing data lends itself readily to regression and some to classification.

Thanks.

[hurrikane]

I find it interesting that you start a thread asking a question but in all of your answers you talk about your NN that you already use
Why don't you give the stats of the system, data source, theory, etc. Or even a part. you asked for ideas but seem to be trying to validate your system.

Just post the picks and lets see how it works.

As for the 2000, why did you select that?
Is that all burger, sprints, specific tracks, fields, colors, what?

I only ask for reference. Also, do you update your 2000 races daily adding the new and removing the old or....?

thanks,
H

[osophy_junkie]

I approach it as a classification problem. A probability is included with the prediction. I use the probabilities of horses predicted to win to look for overlays. I can image regression could be used to calculate a speed rating or some type of form calculation.

Dividing the races is part of the GA. In what way do you use regression?


Ed

[BHURRICANE]

I find it interesting that you start a thread asking a question but in all of your answers you talk about your NN that you already use
Why don't you give the stats of the system, data source, theory, etc. Or even a part. you asked for ideas but seem to be trying to validate your system.

Just post the picks and lets see how it works.

As for the 2000, why did you select that?
Is that all burger, sprints, specific tracks, fields, colors, what?

I only ask for reference. Also, do you update your 2000 races daily adding the new and removing the old or....?

thanks,
H

I am interested in others experiences in presenting data to neural nets and SVMs in a general. I'm willing to trade or exchange ideas and/or code with others that do the same. I'd be happy to discuss picks in the War Room, and usually do my playing on the weekend. 2000 is not an arbitrary number, I examine about 3 or 4 cards a day and usually pick 2 or 3 races per card to put into my analysis database. The data accumlates about 1800-2200 races per year. The data is segmented by track, distance, race type, conditions, etc. I do retire data as it seems to only have usefulness at some tracks for short periods of time. If when backtesting I find the "retention" of older races at a track are hurting the return, then they are removed.

[BHURRICANE]

I approach it as a classification problem. A probability is included with the prediction. I use the probabilities of horses predicted to win to look for overlays. I can image regression could be used to calculate a speed rating or some type of form calculation.

Dividing the races is part of the GA. In what way do you use regression?


Ed

I use the regression approach to almost all data related to a horse's performance, and across any given time series.

Here's a rough of the program. I'm looking for how others are handling their dataset preps.

+ Dataset Prep

- Removal of contradictory data
- Remove errant data

+ Dataset Analysis

- Partition the dataset into train, validate, test
- Run a GA to select most relevent data

+ Dataset Preprocessing

- Numeric scaling and encoding
- Categorical encoding
- Cyclical encoding of time values

+ Produce network
+ Train network
+ Test network
+ Good network? Keep it. Else --> Trash.

[BHURRICANE]

Just post the picks and lets see how it works.
thanks,
H

1st time and low data horses not considered
AP1 1-1A-5-6-2
AP2 2-3-7-6-1
AP3 1-4-2-7-6
AP4 14-12-5-13-8
AP5 6-7-2-1-4
AP6 8-1-7-6-2
AP7 2-4-8-5-3
AP8 7-12-6-2-11
AP9 1-5-11-6-8
DEL1 1-2-7-4-6
DEL2 6-8-1-7-4
DEL3 4-5-6-7-1
DEL4 3-5-12-11-1
DEL5 13-6-8-14-7
DEL6 4-8-7-3-1
DEL7 4-2-1-5-6
DEL8 5-4-8-6-3
DEL9 8-10-7-2-6
DEL10 2-3-6-5-7
DMR1 1-4-5-6-2
DMR2 2-8-4-6-7
DMR3 8-2-1-4-7
DMR4 9-2-10-7-5
DMR5 9-6-8-3-1
DMR6 4-3-1-2-8
DMR7 3-2-1A-6-1
DMR8 9-5-11-4-2
ELP1 5-2-3-1-4
ELP2 6-3-5-4-2
ELP3 4-6-5-1-7
ELP4 1-6-3-9-5
ELP5 14-6-8-13-11
ELP6 2-10-5-6-7
ELP7 10-1-4-7-6
ELP8 6-7-5-4-1
ELP9 10-9-3-7-6
EVD1 3-5-7-1-2
EVD2 2-6-3-7-4
EVD3 8-4-10-5-7
EVD4 8-5-7-1-6
EVD5 11-4-2-7-12
EVD6 11-1-2-3-4
EVD7 1-6-4-5-3
EVD8 10-9-12-7-6
EVD9 8-7-1-5-6
EVD10 1-10-6-12-5

[hurrikane]

Ok
I"m confused. You have reduced the fields to 5 horses. Is the first horse your top selection. How are you going to bet these?

[BHURRICANE]

Ok
I"m confused. You have reduced the fields to 5 horses. Is the first horse your top selection. How are you going to bet these?

The order is for identifying overlays. If the horses listed near the top post nice odds, then I consider them in an exacta box or win tickets.

[GameTheory]

- Cyclical encoding of time values


Not sure I understand this one. Could you explain what that means exactly?

Also, what is your input / output for a new race? Do you run each horse individually through the network and get an output value for that horse alone, or do you feed it entire races at one time?

[Jaguar]

Used Dave's brilliant "Thorobrain" very profitably for several years.

After he moved on, I did not upgrade to the successor's version simply because of the "data provider" straightjacket which the developer forces on Thorobrain users.

I prefer to use BRIS, where I have had an account for nigh on to 20 years, and which provides inexpensive files(TSN is even cheaper, by the way), which are compatble with a number of programs I use(new and old).

Don't know if RACECOM's "NSPACE" is a neural net platform, because Joe and David Shepard guard their software secrets very carefully.

One day I may just treat myself to the new Thorobrain, just because I'm a compulsive software buyer, and I'm certain I can earn money with it. Also, I'm eaten-up with curiosity to see if Thorobrain is as good as RACECOM's latest version, which I have used.

The barrier to entry into RACECOM's new world is the purchase price, which Joe has bumped to $1,800.

Not saying it's not worth it, it's a superb program. But, I already own good programs and they cost alot less.

One day some sharp programmer is going to write a killer genetic programming app, which will blow us all away.

Meanwhile, having profited from Dave's neural net handiwork, I gotta be a neural net booster- as far as handicapping is concerned.

All The Best,

Jaguar

[BHURRICANE]

Not sure I understand this one. Could you explain what that means exactly?

Also, what is your input / output for a new race? Do you run each horse individually through the network and get an output value for that horse alone, or do you feed it entire races at one time?

Performances may depend on a day of the week or a season...
I encode date columns into two new columns:
new column1 = sin((X/P)*2*Pi)
new column2 = cos((X/P)*2*Pi)
where X is the column value and P represents periodicity Pi is 3.14159,
a weekly cycle would have a periodicity of 7 for example. So that if we encoding a Sunday as 1 and Saturday as 7 the network might be confused that those values are far apart when they are not, this avoids that.

I have a network that forecasts the horses indivdual performances, those outputs are the inputs to the race network which considers all the horses at the same time.

[GameTheory]

Performances may depend on a day of the week or a season...
I encode date columns into two new columns:
new column1 = sin((X/P)*2*Pi)
new column2 = cos((X/P)*2*Pi)
where X is the column value and P represents periodicity Pi is 3.14159,
a weekly cycle would have a periodicity of 7 for example. So that if we encoding a Sunday as 1 and Saturday as 7 the network might be confused that those values are far apart when they are not, this avoids that.Ahhh... I had never thought of doing that. But the only period I use is the year, and simply use DAY OF THE YEAR (1- 365) for dates. Since there is a vast difference (due to the universal January 1 birthday) between December 31 and January 1, I think I'll stick with what I've got.

I have a network that forecasts the horses indivdual performances, those outputs are the inputs to the race network which considers all the horses at the same time.And so there are multiple outputs to this second race network?

[hurrikane]

[QUOTE=BHURRICANE]The order is for identifying overlays. If the horses listed near the top post nice odds, then I consider them in an exacta box or win tickets.[/QUOTE


B-
this seems very amibiguous. After you reduce the fields to 5, which is nothing magical, you then in some arbitrary way select a horse to bet?

This is the problem I have had with NN in general. They will get you so far and then you are stuck. i can reduce fields to 5 horses and get a 90% hit rate but that doesn't make me money.

Any ideas on how you are going to assign value? What are 'nice odds'? My initial thougths were that a NN could assign a MLO based on risk assesment much like an actuarial might do for an insurance company. I just never got it to do that in a profitable way. But I haven't given up.

[NoDayJob]

One day some sharp programmer is going to write a killer genetic programming app, which will blow us all away.
Jaguar

I dare say, "It's been done already.", many times, but they will never be sold commerially, because the programmer[s] are making lots of money and keeping it to themselves.

NDJ

[Maxspa]

All,
Just received the Handicapping Times for BRIS. The freebie for the month of August is 3 FREE Neurax Pro files. Supposedly the Free Software Neurax Pro has been updated. I haven't used the program and reports here have not been favorable, but for someone interested in trying neural networks this could be a fun experience.
Maxspa

[CapperLou]

Hope you are doing well Jag!!! As you may recall--I have not used TB5 in almost two years. There were some nice winners--but as I told you once I could not separate out the ones that were pointed to in different ways to set up a pattern of play that was profitable. I had a lot of fun with it and hit some mighty big ones, but overall it did not work for me and believe me I tried everything. Neil was very helpful too. He's a great guy!!

All the best,

CapperLou

[GameTheory]

All,
Just received the Handicapping Times for BRIS. The freebie for the month of August is 3 FREE Neurax Pro files. Supposedly the Free Software Neurax Pro has been updated. I haven't used the program and reports here have not been favorable, but for someone interested in trying neural networks this could be a fun experience.
Neurax is a complete joke and should be avoided, especially for those interested in neural networks as it will leave an unfavorable impression. A better name for Neurax would be "Backfitter Pro".

[Que]

BHURRICANE,

This is certainly an interesting thread. Although I once invested a rather significant amount of time researching a few of these methods, I've recently been spending the majority of my time conducting a comprehensive study of many of these same techniques on the financial markets. The application of these approaches is very similiar, albeit the data preprocessing is totally different. Although time series analysis may not be applicable to horse racing, but pattern recognition techniques certainly are. However, it appears that you've taken these approaches to handicapping much further than I ever did. My only advice is to make sure that your data is properly balanced, otherwise your network or classification algorithm may only learn what's in common with all the loser's and it won't learn much, if anything, about the significant attributes of the winner. However, if you are using a separate network or algorithm to post-process your intial results then this might not be a major problem. Also, I think it would be possible to apply various qualitative and limited dependent variable models (QLIM) to handicapping, but I've never tried it. These QLIMs could include linear, probit, or logit models. Theoritically, if you output the probability of all the responses, i.e. the probability for each horse to finish in each position, then you could have the start of an odds line. Although how effective this odds line is would require some substantial testing. You could also use the outcome of this model as an input to another model. For example, the dependent factor of your first model could be position at the 1st call, 2nd call, etc. By combining several of these models, your final model could be more robust.

Anyway, my previous experience with handicapping has been to use a hybrid neuro-genetic algorithm to create a race rating that incorpartes the time at each of the horse's pace calls. The "pace" weights asssigned to each call are specific to each track and/or distance. I still use this method when I compile my 3YO horse rankings prior to the Kentucky Derby each year. I've also used decision trees and classification and regression trees (CART) to analyze trainer patterns. This later method had limited success (it was much better than the track take, but didn't show a flat bet profit); however, I failed to properly balance the data prior to running the classification algorithm so the study was flawed at the start. Whenever I have the time I'll try to rewrite the program to properly preprocess the data. One concern I have with your approach, is the post-selection of races to include in your training sample. While this approach could certainly improve your percentage of winners, it seems to me that it might skew any attempts to create an accurate odds line afterwards. I'm not sure if this would actaully be the case, but I think the likelihood is high enough to warrant further research.

I also like the way you incorporated "periodicity" into your research. I've been wanting to incorpoarte this feature into my financial research for some time, but I didn't like the way most people incorporated periodicity into their time series data, i.e. by adding a separate data column for each day-of-the-week, etc.

Anyway, best of luck in your search.

Que.

[GameTheory]

My only advice is to make sure that your data is properly balanced, otherwise your network or classification algorithm may only learn what's in common with all the loser's and it won't learn much, if anything, about the significant attributes of the winner.
Balance the data in what way? Please elaborate.

I have tried zillions of ways of pre-processing the data in terms of what races/running lines to leave in or take out, and always keep coming back to just leaving the sample as it is, and letting the distribution fall where it may...

[hurrikane]

Have any of you done any work with boosting algorithms to do the weighting?

I've just started looking at this but is seems viable. there have been a couple of applications do on horse racing from down under using svm and boosting.

any opinions or advice would be greatly appreciated.

I'm also applying this to financial markets so any opinions that are different on that end would be great.

[Que]

GameTheory,

What I mean by balancing, is to "correct" your training data set so that the outcome (dependent variable) is more equally distributed. For example, suppose your data set has only two outcomes--win or lose--and in 90% of your training cases the horse lost, while only 10% of the horses in your training set won. Many modeling techniques, especially neural networks, have trouble with such biased data because they will tend to learn only "why" a horse lost and and ignore why a horse won since it is more rare. If the data is well-balanced with roughly equal numbers of winners and losers, many models will have a better chance of finding patterns that distinguish the two groups.

Hurricane,

If you are refering to modeling using decision trees and rulesets then the answer is yes--I have experimented using boosting techniques. Just to be clear, I am defining boosting as the building of multiple models in sequence. The first model (tree) is built the usual way. Then a second tree is built in such a way that it focuses on the records that were misclassified by the first tree. Then a third tree is built to focus on the second tree's errors, and so on. Finally, the records are classified using a weighted voting procedure to combine the separate predictions into one overall prediction. Obviously, boosting can improve the accuracy of a model during the training phase, but I'm less convinced that it will improve the overall accuracy of the model on unseen data. It's like deciding between a model that favors accuracy vice a model that favors generality. In addition, my cardinal rule when developing models, financial or otherwise, is that I don't care how a model performed on the training data. For example, I don't care if a model had a 90% ROI or a 4% ROI on the training data. If given enough variables, any model trained on past data can obtain perfect accuracy--therefore, the only results I care about is how the model performed on a walk-forward basis or on the unseen (future) data. I'm also a firm believer that the fewer variables the model uses the more robust the model is likely to be in the future--therefore I use the Akaike information criteria and/or the Schwarz criterion to evaluate my models. As you probably already know, these information criteria will penalize a model based on the number of estimation coefficents it uses, i.e. the more variables in the model the higher the penalty. For this reason alone, I'm cautious when using boosting techniques.

Que.

[GameTheory]

I've found all these sorts of adjustments are very technique-dependent. Balancing the data might be needed with NN's, but not be helpful with some other technique. It seems to me that a person wishing to do computer modelling for horse-racing really needs to pick what methods they'd like to use first (neural networks, genetic algorithms, standard regression, decision trees, SVMs, hybrids or ensembles of any or all of these), and then figure out what they need to do with the data to make it work for that particular method. I've tried all of them and I believe the data needs to be prepared differently for each, and certain methods will mix well with others and others won't.

We talked about this a bit in an old thread:

http://www.paceadvantage.com/forum/showthread.php?p=92246&highlight=non-trivial#post92246

[hurrikane]

thanks guys.

So I'm curious how do you then decide what data goes in and what stays out?

This is where I always end up. Justifying what stays and what goes.

[osophy_junkie]

I don't see the benifit to filtering by anything you don't know before hand (eg surface type, surface condition, racing conditions, est pace line, etc..). Assuming the data is correct all of it should go into making your model. You don't have the luxury of forsight when betting for real, why rely on it when backtesting?

The most profitable way to split up your data varies from track to track and time of year.

[GameTheory]

I don't leave anything out. Of course I don't train with every race from all tracks, but I just create datasets from broad categories (like track, sprint/route, dirt/turf, etc). Techniques like bagging (training on multiple subsamples drawn from the main sample) lessen the weight of rare events/situations without eliminating them altogether. Of course bagging only works well with some modelling methods. (Bagging doesn't work very well with logistic regression and with some types of neural nets, for example.) So again, you need to pick your method and then figure out the best way to get it working well...

[hurrikane]

GameTheory,

What I mean by balancing, is to "correct" your training data set so that the outcome (dependent variable) is more equally distributed. For example, suppose your data set has only two outcomes--win or lose--and in 90% of your training cases the horse lost, while only 10% of the horses in your training set won. Many modeling techniques, especially neural networks, have trouble with such biased data because they will tend to learn only "why" a horse lost and and ignore why a horse won since it is more rare. If the data is well-balanced with roughly equal numbers of winners and losers, many models will have a better chance of finding patterns that distinguish the two groups.

Que.

Agreed Game,
I guess i am questioning Que's assertion of balancing the data.
Que, how do you decide what data stays and what goes to make a balanced dataset. Since in any giving race there are 5 - 12 times more losers than winnners who stays on the losijng end and why?

[GameTheory]

Agreed Game,
I guess i am questioning Que's assertion of balancing the data.
Que, how do you decide what data stays and what goes to make a balanced dataset. Since in any giving race there are 5 - 12 times more losers than winnners who stays on the losijng end and why?Balancing works well sometimes. (The vanilla version of boosting assumes a balanced mix of positive and negative examples, btw.) If you use an ensemble technique (multiple models trained on different, usually overlapping subsamples), you might just oversample the winners without leaving any of the losers out. I have trained ensembles consistings of *tens of thousands* of individual models with each training sample consisting of exactly two training examples -- one winner and one loser. (The logic being that at the most basic level what we want is for our models to be able to separate winners from losers.) This naturally makes each model extremely simple, but the ensemble makes up for that to some degree. This method actually works quite well (sometimes) at major tracks where "primary" factors are more formful. It doesn't work at all at places like Mountaineer where a more complex examination is usually called for. (I think this method probably yields similar results to simple linear weighting of each factor.)

That brings up another tangent -- with non-statisical methods like neural networks, the bigger the sample size, the more complex the resulting model will be, sometimes too complex. So pruning the dataset is advantageous in that case, you just have to figure out how to prune it. I don't think there is one right way to do it. You have to choose what you want to focus on -- what question do you want the model to answer?

[hurrikane]

Thanks GT.

appreciate the input. I haven't found the golden nugget in all of this yet but this helps.

Curious if you have done any modeling on betting with the curve.

when the win % is higher than expected reduce bets, when lower increase. going againt the trend to play to regression to the mean.

I have done some work here without great results but thought you might have gone a little deeper.

[GameTheory]

Thanks GT.

appreciate the input. I haven't found the golden nugget in all of this yet but this helps.

Curious if you have done any modeling on betting with the curve.

when the win % is higher than expected reduce bets, when lower increase. going againt the trend to play to regression to the mean.

I have done some work here without great results but thought you might have gone a little deeper.The golden nuggets can only be found by experimentation with your exact methods. You can read all you like to get ideas, but you won't know what actually works until you try it.

As far betting on the curve, I have been tempted to, but since logic and probability theory dictate that betting on things that are "due" is a fool's game I have not actually tried it. In other words, in theory it shouldn't work even if it sounds good. If you flip a fair coin 10 times in a row and they are all heads, it is still a 50-50 chance that the next one will be tails. Human brains have trouble understanding that and probability in general -- I believe there was some recent discussion about the "Let's Make a Deal" problem on this board recently, and most people came up with the wrong answer. The wiring in our brain has trouble comprehending probability properly...

[hurrikane]

I agree with the probability side of things. Learned a long time ago there is no such thing as 'due'.

I have noticed recently in mining some factors lose some juice during certian parts of the year. In fact I may even take the winter off this year or only play a couple of southern tracks. I don't think that just because a play has not been winning I should stop betting it or even slow down. In fact one of my biggest scores came right off of a 30 race losing streak.

So, I don't believe in due column, or anything like it. But I am wondering if, as tracks are sealed, trainers and jockies get ready for the tp and bc. and other factors, can have an effect. Even drought in the midwest affecting the feed.

I dont' know. sometimes I look where mayber there really are no answers.

[osophy_junkie]

... Techniques like bagging (training on multiple subsamples drawn from the main sample) lessen the weight of rare events/situations without eliminating them altogether...

Do you retrain with the same information? Retraining at more precise time intervals might be a good way to address the issue of different scale. (eg. Initial traning uses data calculated from the prior 3 months. retraining happens with data 1 month prior, then again with 2 weeks prior). The idea being to remove noise by segmentation instead of by volume.

[GameTheory]

Do you retrain with the same information? Retraining at more precise time intervals might be a good way to address the issue of different scale. (eg. Initial traning uses data calculated from the prior 3 months. retraining happens with data 1 month prior, then again with 2 weeks prior). The idea being to remove noise by segmentation instead of by volume.With bagging, you train on bootstrap replicates of the main sample. Let's say your main (total) sample has 1000 examples in it. To create a bootstrap replicate sample, simply pick randomly from the main sample 1000 times. You do this "with replacement" meaning you go ahead and pick some examples twice or more. About 1/3 of the main sample will not make it into the training sample because of the repeats. You make lots of these training samples and train a model with each one. Then at prediction time, you take the average output from all the models put together.

[Que]

Hurricane,

First, I'd like to say that I agree with everything GameTheory has posted in this thread. He obviously has alot more experience in this area, and IMHO his comments appear to be right on the mark. Also, as GameTheory mentioned earlier, balancing certainly won't work for all modeling techniques. However, it is one of first things you might want to consider if you feel your results are unsatisfactory. But just like any other technique--the only way to find out if a method works is to experiment with it.

With regards to your question on deciding "what data stays and what goes to make a balanced dataset," I don't think there is a single right way to do it. But if you decide to balance your data you have two choices, i.e. you can either oversample the winners or undersample the losers. Both methods have own their drawbacks. For example, if your starting dataset consists of 10% winners and 90% losers, by oversampling you would need to replicate your winning records by a factor of nine. However, this method could create other bigger problems. If you are analyzing trainer patterns, an example of this might occur if your original sample contained one infrequent trainer/jockey combination which had two wins in two starts. If you oversample, this specific jockey/trainer combination has now changed from 2-for-2 to 18-of-18--which is very very unlikely. If your modeling method then discovered this unrealistic combination it would be to the detriment of more important factors.

To avoid the potential problem described above you could undersample your losers instead--which brings me back to your original question, i.e. "what data stays and what goes to make a balanced dataset?" One way to do this would be to divide your dataset of losing records into separate clusters. For example, if you had 100 winners and 900 losers, you could use kmeans clustering to divide your dataset into 9 separate clusters. Then you could randomly choose an equal number of records from each cluster. For example, your balanced dataset would then consist of your original 100 winners, and 11 randomly selected records from each of your 9 clusters--plus one randomly chosen record from the entire dataset, so that the number of losing records now equals the number of winning records.

As I stated at the beginning, there's no single right or wrong way to do this. Personally, I would prefer to undersample the data using a method like I described above. However, the only way to find out which method works best for this type of model, designed for this specific purpose is through experimentation. You might discover that oversampling is much quicker and just as effective as undersampling. You might even discover that you don't even to need to balance your data at all.

Que.

[traynor]

Game Theory wrote: <With bagging, you train on bootstrap replicates of the main sample. Let's say your main (total) sample has 1000 examples in it. To create a bootstrap replicate sample, simply pick randomly from the main sample 1000 times. You do this "with replacement" meaning you go ahead and pick some examples twice or more. About 1/3 of the main sample will not make it into the training sample because of the repeats. You make lots of these training samples and train a model with each one. Then at prediction time, you take the average output from all the models put together.>


Random sampling with replacement, using a bootstrap algorithm, selects out slices of the dataset. The key factor in using the technique is the similarity of the samples extracted; if the 1000 sample slices are relatively consistent (meaning within 10% of each other) your model is stable--and predictive. If 90% of the sample slices vary from each other by more than 10%, all you have is a lot of essentially meaningless data.

The bootstrap technique can be extremely useful, particularly at eliminating the silly outliers that so many "researchers" want to keep in to fluff up results. It can also be useful to determine when a data model is predictive. Again, 90% of your dataset should be with 10 percentage points, or it is massaging numbers in the hope of imitating science.

We use the bootstapping algorithm (a very simple add-in to Excel that anyone can use) extensively for wagering data models. That said, the biggest argument against bootstrapping is that it creates the illusion of certainty where none exists--if the dataset is biased, the bootstrap will replicate the bias. Mix a few outliers into the model, and what comes out is useless. At least for betting in the real world (as opposed to academic research, or marketing campaigns).

For betting purposes, we routinely truncate the low 5% and high 5% of mutuels, then run a bootstrap on the remaining 90%. It works. It makes money. If you want to argue against the logic, run the algorithm on your own dataset--it can be a real eye-opener.
Good Luck

[traynor]

Que wrote: <To avoid the potential problem described above you could undersample your losers instead--which brings me back to your original question, i.e. "what data stays and what goes to make a balanced dataset?" One way to do this would be to divide your dataset of losing records into separate clusters. For example, if you had 100 winners and 900 losers, you could use kmeans clustering to divide your dataset into 9 separate clusters. Then you could randomly choose an equal number of records from each cluster. For example, your balanced dataset would then consist of your original 100 winners, and 11 randomly selected records from each of your 9 clusters--plus one randomly chosen record from the entire dataset, so that the number of losing records now equals the number of winning records. >

Interesting problem. If you have a 1 to 10 ratio, it would be interesting to slice segments of your dataset (sampling with no replacement) to see how stable that ratio is over time, and how big a sample is necessary for it to be stable. Averages are often wildly misleading as anything other than averages.
Good Luck

[Dave Schwartz]

I use a similar but slightly different approach to "bagging."

Using the aforementioned example of 1,000 races, I would draw each race at random until I had accumulated 10,000 races in the sample. Each time I draw a race, I add a degree of randomness to the finish, thus adjusting the outcome.

This way the horse that wins by a nose or a neck (or more - depending upon the degree of randomness) does not always win and the system learns not to depend upon that race to justify itself.


Regards,
Dave Schwartz

[GameTheory]

Random sampling with replacement, using a bootstrap algorithm, selects out slices of the dataset. The key factor in using the technique is the similarity of the samples extracted; if the 1000 sample slices are relatively consistent (meaning within 10% of each other) your model is stable--and predictive. If 90% of the sample slices vary from each other by more than 10%, all you have is a lot of essentially meaningless data.What does that mean exactly? How do you determine if one sample is 10% away (or whatever) from another? If you have a sample row of multi-dimensional data (say 100 variables representing a horse in a race) and compare it to another row (the same 100 variables for a different horse), by what method do you determine how many percentage points one is away from another? There are various "distance" metrics for that sort of thing, but saying one row is X% away from another in a multi-dimensional space doesn't make sense to me.

To me, the whole point of using something like bagging is you often don't know what the outliers are because of the dimensional complexity of the problem. I also don't see how bagging could take a "meaningful" sample and turn it into a "meaningless" one as you seem to imply might happen in certain cases. The worst that usually happens if you use bagging with a learning method that won't benefit from it is that it just doesn't help, or you will end up with *slightly* worse results than if you had just used the set as-is...

[Charlie Judge]

I have done a lot of work with neural networks of various architectures, genetic algorithms, and bootstrapping. I have never found a NN which outperformed a database based statistical lookup method. I use the NeuroShell2 system from Ward Systems group in Frederick, MD ( an old package) and have found that is excellent at backfitting, but not better than mra or stats for forward prediction.

One advantage of the neuroshell package is the output of a c language, visual basic, or pure formula output to build the net for use in any other program. This allows you to incorporate the nn in your program without having to shell out to a separate nn package.

I look forward to trying some of the suggestions mentioned in this thread, but so far, no outstanding nn results above my own stats so far.

caj

[traynor]

Dave Schwartz wrote <Using the aforementioned example of 1,000 races, I would draw each race at random until I had accumulated 10,000 races in the sample. Each time I draw a race, I add a degree of randomness to the finish, thus adjusting the outcome.

This way the horse that wins by a nose or a neck (or more - depending upon the degree of randomness) does not always win and the system learns not to depend upon>

This is exactly the argument used against bootstrapping. With random sampling without replacement, the 1000 race sample has "expanded" as if it were 10,000 races. It is still only 1000 races. The basic algorithm seems reasonable and logical. Used carefully, it can be. Used incorrectly, it produces mountains of data that essentially repeat the base data--and any errors or logic flaws in that 1000 race sample are simply magnified.
Good luck

[traynor]

GameTheory wrote <What does that mean exactly? How do you determine if one sample is 10% away (or whatever) from another? If you have a sample row of multi-dimensional data (say 100 variables representing a horse in a race) and compare it to another row (the same 100 variables for a different horse), by what method do you determine how many percentage points one is away from another? There are various "distance" metrics for that sort of thing, but saying one row is X% away from another in a multi-dimensional space doesn't make sense to me.>

We are talking about horse races, not multidimensional space. The bootstrapping model--random resampling without replacement--simply generates more data to create the illusion of a larger sample. Random sampling of a truncated dataset (at 90%) "slices" that dataset into essentially similar groups. If the groups are not similar, there are aberrations that are corrupting the results.

If I break a 2000 race sample into 20 samples of 100 races each, I expect them to be within 10% of each other in win%, average mutuel, and ROI. If not, I want to know why. In a 10,000 race truncated sample (the model we normally use for wagering), rather than simply accept the results, we again truncate the top and bottom 5%.

Specifically, because we are betting on the model, we have a totally different perspective from researchers attempting to prove the predictive capability of their methods or software. The latter are perfectly justified in including outliers, because that is the intent of the research--to demonstrate what the method/software did, or could have done. In wagering, it is far more important to weed out anomalies that are unlikely to be replicated, at least in a reasonably finite period.

CJ remarked awhile back that he had "literally hundreds" of (high priced mutuels) in his records. I have no doubt whatsoever that he is being completely honest. In a sample of 20,000 or 30,000 races, a couple of hundred outliers represent little more than semantic noise. We prefer to eliminate that semantic noise as thoroughly as possible.

Again, it is only because our focus is on wagering; we do not handicap as intellectual enrichment or as a hobby. With that goal in mind, we tend to be somewhat abrupt in tossing unrepresentative outliers as irrelevant.
Good luck

[sjk]

Who are all of these researchers? What would be the point of developing handicapping software unless you used it to bet and win money?

I wonder if the truncated and small sample sized data-sets are accurate in predicting the lower probability events. I often bet exactas that I only expect to win 1 out of 100 times. Just bet one in my last race today that I gave a 1 in 100 chance but since it was paying $914 if it got there it was well worth playing.

[GameTheory]

We are talking about horse races, not multidimensional space.Horse races represented numerically as a series of variables (I assume). If that is true, then you are talking about multidimensional space -- each variable in a example is a dimension. You did not explain in your previous post that you were referring to win%, avg mutuel, and ROI.

The bootstrapping model--random resampling without replacement--simply generates more data to create the illusion of a larger sample.Not exactly, because that implies that you create a larger sample and then train a single model on it. With bagging, you create many bootstrap samples the same size as the original sample and then create a separate model from each sample. (Using half-size subsamples without replacement produces similar results.) Then you combine the output of the models. The point is that it is *OK* for the samples/models to be different because those differences are smoothed out by the combining process, leaving you with the commonalities between models.

You eliminate races that have rare payoffs, but what about races with normal payoffs where the winner won for a rare reason? Outliers in any dimension of the multidimensional space (which we are talking about, sorry) are automatically smoothed out with a technique like bagging. But to repeat myself once again, bagging (or similar) may not work with all learning methods. In my experience, it doesn't work well with neural networks because they are too stable. It works best with learning methods that are unstable -- meaning slight differences in the training sample will cause the created models to be quite different.

[traynor]

GameTheory wrote <Not exactly, because that implies that you create a larger sample and then train a single model on it. With bagging, you create many bootstrap samples the same size as the original sample and then create a separate model from each sample. (Using half-size subsamples without replacement produces similar results.) Then you combine the output of the models. The point is that it is *OK* for the samples/models to be different because those differences are smoothed out by the combining process, leaving you with the commonalities between models.>

All the way back to the origin of the bootstrapping concept--used to predict crop yield in India--the argument against it is that "you are trying to get something for nothing." Not my argument, but that of people far more knowledgeable in the statistics field, including a number of stat and math PhDs teaching MBA classes. The bottom line is usually, "Yes, very impressive ... but what do these results indicate?" That generally precedes a very embarassing dissection of flawed logic in front of an amused (and very large) class.

The underlying logic of bootstrapping is seriously flawed, and the technique can only be used meaningfully with serious modification. "Multidimensional space" is not a particularly useful description; like bootstrapping, it implies a connection to serious research that may be illusory.
Good luck

[traynor]

GameTheory wrote <With bagging, you create many bootstrap samples the same size as the original sample and then create a separate model from each sample. (Using half-size subsamples without replacement produces similar results.) Then you combine the output of the models. The point is that it is *OK* for the samples/models to be different because those differences are smoothed out by the combining process, leaving you with the commonalities between models.>

Assuming you are serious, the problem we have with the procedure is similar to the one we have with a number of other "sophisticated decision models"--they generate nice charts, graphs, and numbers, but signify little. If you have a 1000 race sample, all possible results of your model are already in that sample; expanding it to "simulate" 20,000 races adds nothing, and subtracts nothing.
Good luck

[Dave Schwartz]

Traynor,

My goal in creating the 10,000 race sample out of the 1,000 race sample is to try to build a system that is more general than specific.

My experience with neural nets is that their very nature causes them to demand perfection of themselves. That is, when you consider a 10% thereshhold the following statement applies:

A neural net is trying to be 90% right 100% of the time.
That is, its "expectation" is to get all the winners.

As a horse player, I need a system which is trying to be 100% right, 90% of the time.
In other words, I want a system that says, "Hey, there are some races I just can't get and that's okay."

This is why I left neural nets behind and moved to genetic algorithms. IMHO, they are much better suited to Complex Adaptive Systems (CAS) such as horse racing, commodities or the stock market.


Regards,
Dave Schwartz

[GameTheory]

Assuming you are serious, the problem we have with the procedure is similar to the one we have with a number of other "sophisticated decision models"--they generate nice charts, graphs, and numbers, but signify little. If you have a 1000 race sample, all possible results of your model are already in that sample; expanding it to "simulate" 20,000 races adds nothing, and subtracts nothing.They generate nice charts, graphs, numbers, and extra dollars in my pocket -- you forgot one. I can see you are not interested in discussion, only in sounding superior (as usual), so I won't pursue this further, but I'll point out that I never implied the technique added or subtracted anything -- it helps *distill* the meaning of what is there, just as you feel removing the extreme payoffs from your samples helps you to do. Get it?

[traynor]

sjk wrote: <

Who are all of these researchers? What would be the point of developing handicapping software unless you used it to bet and win money?

I wonder if the truncated and small sample sized data-sets are accurate in predicting the lower probability events. I often bet exactas that I only expect to win 1 out of 100 times. Just bet one in my last race today that I gave a 1 in 100 chance but since it was paying $914 if it got there it was well worth playing.>

Many of the researchers are "professionals" who realize their methods cannot return a profit, so sell them instead. Take a long, hard look at Trackmaster and All-Ways, especially the techniques of "adjusting the software" so that ... if you had adjusted it yesterday, it would have predicted the longshot that won.

We spent more than $15,000 on software apps, then reverse engineered them to determine the algorithms used. It is a common technique in software development, and perfectly legal. In fact, there are "reverse engineering apps" that will automatically deconstruct a program and generate everything but the name of the developer's cat or dog. Bottom line: not one of the apps had algorithms worth using.

In general, the apps use simplistic sixth-grade arithmetic to generate "ratings"--one of which usually "predicts" the winner because each "predicts" a different winner.

Added to the mix is the problem that any deficiencies of the app are tossed onto the user, using the Sartin formula of "You didn't isolate the true contenders ... you didn't select the "proper" paceline." In truth, if you can isolate the true contenders and pick the proper pace lines, the software is redundant--you already know which entry will win. The software is a crutch, and, as far as the Sartin Methodology is concerned, always has been.

As for exactas, I wish you luck with your 1-in-100 model. If you have a 2000 race model and exacta zz won 20, returning a 100% profit, the bottom line question is; What does it mean? It means that in that specific segment of races, exacta zz won 20 times. It doesn't "mean" anything else, and definitely does not mean that in the next 2000 race segment it will win 20 times and return a 100% profit. It could every bit as easily win 4 or 5 times, and post a 70% loss.

Again, this is not opinion. Our research is for wagering, not gathering statistics; it is essential that we be able to separate predictive results (that we can make money with) from interesting results (that many others seem to package for re-sale in "self-adjusting" software that is always late, and always adjusts itself so that it would have won if it had "self-adjusted" before the race, rather than after).
Good luck

[sjk]

Traynor,

I use a 370,000 race database; not that it has to be that large but I would wonder whether the 2,00 or even the 10,000 race samples are large enough to adequately measure the low probability events.

Practical experience along with back-testing gives me confidence that when I get the right price on what I expect to be a 1 in 100 event I have a positive expectation.

I had no idea you were talking about those software vendors when you talked about researchers. Never crossed my mind that they were doing any serious research other than on how to best sell their wares.

[traynor]

GameTheory wrote <They generate nice charts, graphs, numbers, and extra dollars in my pocket -- you forgot one. I can see you are not interested in discussion, only in sounding superior (as usual), so I won't pursue this further, but I'll point out that I never implied the technique added or subtracted anything -- it helps *distill* the meaning of what is there, just as you feel removing the extreme payoffs from your samples helps you to do. Get it?>

My intent is neither to demean your efforts or opinions nor in "sounding superior." I am more than willing to listen to, try, experiment with, or discuss anyone's opinion. I am a professional handicapper, first, last, and always. That means that anything I think will give me an edge is valuable to me.

However, I am not interested in "proving" theories, or engaging in long, rambling discussions that go nowhere. I have come to realize that the overwhelming majority of handicappers are hobbyists; they don't bet, or bet rarely, or bet trivial amounts, because they realize that their betting has a negative expectation. Some of that group are the most vocal, and the most intent on persuading others to adopt their point of view.

I am not suggesting that you fit that category, nor that your concepts are not valuable. I am suggesting that what some may interpret as my desire to "sound superior" is simply a high level of frustration with people who seem unwilling to even consider the fact that their decision processes--the processes themselves, not just the decisions--may be seriously flawed.

Those flaws may be acceptable in research, or, in many cases, even in business. However, if your income depends on the outcome of literally hundreds of your decisions on a daily basis, it tends to alter perspectives. It becomes essential to refine the decision making process to coincide with reality in the future, not to say "I woulda had that one."
Good luck

[sjk]

As for exactas, I wish you luck with your 1-in-100 model. If you have a 2000 race model and exacta zz won 20, returning a 100% profit, the bottom line question is; What does it mean? It means that in that specific segment of races, exacta zz won 20 times. It doesn't "mean" anything else, and definitely does not mean that in the next 2000 race segment it will win 20 times and return a 100% profit. It could every bit as easily win 4 or 5 times, and post a 70% loss.

Again, this is not opinion. Our research is for wagering, not gathering statistics; it is essential that we be able to separate predictive results (that we can make money with) from interesting results (that many others seem to package for re-sale in "self-adjusting" software that is always late, and always adjusts itself so that it would have won if it had "self-adjusted" before the race, rather than after).
Good luck

What you suggest would be a thouroughly inane way of determining that an event happens 1 in 100 times. Give us a little credit.

[traynor]

sjk wrote <I use a 370,000 race database; not that it has to be that large but I would wonder whether the 2,00 or even the 10,000 race samples are large enough to adequately measure the low probability events.

Practical experience along with back-testing gives me confidence that when I get the right price on what I expect to be a 1 in 100 event I have a positive expectation.>

You certainly have a large enough database, and if you are getting probabilities using that base, they may be reasonably accurate. My thought is that there is a mid-range somewhere that is large enough to be predictive, yet small enough to be meaningful.

With a very large database, it seems the main danger is combining too many results that are unrelated to today's event; for example, a monster exacta last year at Arapahoe Park or Atokad Park or wherever, mixed with results from Aqueduct. It seems more useful to use a smaller dataset, more tuned to the specifics of today's event. Again, as example, comparing turf races at Belmont with a dirt races at Aqueduct. There are similarities, yes, but the dataset is "fuzzy."

Since Quirin's 1977 study, lots of people have been searching for "universal truths" that apply to (for example) six furlong sprints at every track, in every season, and at every grade. Technically, that approach suffers from a lack of focus, and a conceptual deficiency in the underlying "logic" of statistics. If you are making money with it, congratulations on a job well done.

We have found it essential to focus research in the specific areas we wager on, using specific models, rather than generic. We also use a substantially different process for calculating results--in essence, every race is handicapped, the results recorded, then analyzed (by the app) based on results. Regression analysis, even multiple regression, was discarded early as unproductive. The regression analysis we use is of our own handicapping, rather than of simple historical data.

The reasoning is simple; there is no point in including "wins" after the fact that the algorithms in the application we use would not have predicted before the race. That seems to be the greatest deficiency in "research" on handicapping; second-guessing reality after the fact and believing "I woulda had that one."

Another problem we encounter is that virtually every time we make the slightest modification to the algorithms, it tweaks all previous results. That means that if we change an impact value from 1.23 to 1.25, we have to literally re-handicap every single race in the dataset, using the "new" algorithm, to adequately compare the results of the change. To give a simplistic example, the "new" algorithm would have included a $4 winner, but excluded a $26 winner in the same race predicted by another value.

We have found no simple way to meaningfully analyze the dataset--if algorithms are changed in any way--unless we completely re-calculate every race in the dataset using only the information that was available before the race was run. Similarly, if there is the slightest change in the process used to select pacelines, or to include or eliminate entries as contenders, every race in the dataset has to be re-calculated.

The alternative is a meaningless model. Specifically, if a race is included that is based on "intuitive" information processing unique to a specific race, that may or may not be replicated in a re-calculation of that race at some later date, the entire concept of a "model" is silly. Whether successful or not, the net result is a large number of one-race models.
Good luck

[traynor]

sjk wrote <What you suggest would be a thouroughly inane way of determining that an event happens 1 in 100 times. Give us a little credit>

Win frequency is a big topic. If you have a 370,000 race database, and you are working on a 1-in-100 frequency, there is a reasonable technique to use to find out if it is worth betting, and one that bypasses the obfuscation of using averages.

Use two counters. Set both to zero. Set A to zero every time event Z occurs, and add 1 every time it doesn't. Create a secondary test that tests when A is greater than B, and, when it is, assigns the value of A to B. Then run your entire sample. At the end of it, the value of B will be the longest string of losses.

While event Z may happen an average of 1-in-100, there could be missouts of 1000 or more races. If you collect such a figure, and in 370,000 races, event Z had some manageable number of missouts (200? 300?) maximum, then I would be a lot more inclined to bet on it.

We rarely use averages unless those averages include information on frequency; the "mean" alone is little more than a number, especially when based on a large dataset.
Good luck

[sjk]

Traynor,

I seem to get tempted to make changes to my program once or twice a year. The backtesting process is a painful one; I have to keep the computer running more or less around the clock for a day or two to test what the results would have been for a year's worth of races.

I posted elsewhere that I recently reversed two program changes that I made earlier in the year as I didn't think they were providing the anticipated benefit. There is only a finite amount of data in the data record and the perils of backfitting are real even if you try to calculate things in a very broad way.

It might be a good plan for me to give up on changes and leave things alone for the long haul.

As far as mixing dirt and turf races, I don't look at turf races at all as they are fundementally different. I find that a single program can adequately cope with differences in distance and track as long as the right compensating elements are in place.

Thanks for your words of congratulations.

[sjk]

I have no event Z in mind that has a 1 in 100 expectation. My example related to a particular bet that I had just made and I could have equally well used any of a hundred other bets that I had made that day, each with its own probability. I would think that hunting around for specific events that appear to have a predictable frequency would be full of pitfalls.

I stick with very general calculations. I calculate the probabilities of the horses running 1st or 2nd and the associated exacta probabilities and bet the overlays.

It has been many years since I concerned myself with whether my overlays were worth betting; I just make the bets.

[traynor]

sjk wrote <I have no event Z in mind that has a 1 in 100 expectation. My example related to a particular bet that I had just made and I could have equally well used any of a hundred other bets that I had made that day, each with its own probability. I would think that hunting around for specific events that appear to have a predictable frequency would be full of pitfalls.

I stick with very general calculations. I calculate the probabilities of the horses running 1st or 2nd and the associated exacta probabilities and bet the overlays.>

That clears up a lot! I have heard any number of handicappers use the "once every hundred races" theory because a regression indicated the proverbial event Z had happened 20 times in 2000 races. I included the loop we use in regressions (recently added) that generates a "longest string of losses" in the entire dataset. It is an extremely useful piece of information.

I agree completely with your evaluation of "updating" as being a very time consuming, computer intensive process. I think that is one of the biggest errors researchers make, and one of the strongest advantages a serious handicapper can exploit.

I don't know what kind of computing power you are using, but we are running tests on two very promising setups; the first involves "grid computing," a spinoff of parallel processing, that uses four to six desktops working in concert. The second is a Beowulf configuration, that essentially does the same thing--it "links" four to six desktops into a single entity. The networking setup is easy; the programming necessary to take advantage of the multithreading, parallel computations of a number of different nodes is much less so.

The major problem, as you know, is that the calculations must provide only the information that would have been available to the application before each race. Just programming the different date formats was a nightmare, until we scrapped all the formats in favor of a five digit number representing the day of year (three digits) and last two digits of year. That made comparisons of race dates and pacelines a LOT easier!

You say you calculate probabilities of a horse running first or second. I would suggest you might find it worthwhile to look at how often your top rated entry (in whatever category) places. There are a number of very profitable patterns, especially at NYRA and Florida tracks, that use the "normal" top choice to win, coupled with the highest ranking in another (and sometimes useless for win betting) category of rating. That is, the highest ranking in the rating used for place modeling may place substantially more than win, and so be largely ignored or underbet.

Old technique of "contrarian" or "opposite" ratings was similar; bet top speed to win, top class to place, or top early pace to win with top sustained pace to place, and so on. Most bettors seem to fixate on "the numbers" and naturally assume that if they are betting the highest rating in a given category to win, the second highest ranked entry in that same category is the best bet to place. That is very rarely the case.
Good luck

[sjk]

Traynor,

Sounds like an impressive hardware set-up. I just use a simple PC although I use a second machine if I have a lot of calculation to do. They are not networked in any way; I just start one on a different month than the other and combine the results later.

I don't really think it is necessary to have hundreds of thousands of races; I did the greater part of my program development years ago when I probably only had a few tens of thousands of races. That was a good thing since the hardware I was working with was so much slower that I don't think it could have handed the current DB size.

[osophy_junkie]

I don't know what kind of computing power you are using, but we are running tests on two very promising setups; the first involves "grid computing," a spinoff of parallel processing, that uses four to six desktops working in concert. The second is a Beowulf configuration, that essentially does the same thing--it "links" four to six desktops into a single entity. The networking setup is easy; the programming necessary to take advantage of the multithreading, parallel computations of a number of different nodes is much less so.

The key to clustering is how you segment the problem being solved. The less time you spend tending network IO the more time the CPU can spend on your problem solving code. While preempitive threads allow rapid prototyping they do not scale well at all. Forking and event driven systems offer much better solutions IMO.

[cosmicway]

Have you included this one in your models ?
What is the best way to handle it ?

[traynor]

osophy_junkie wrote: <The key to clustering is how you segment the problem being solved. The less time you spend tending network IO the more time the CPU can spend on your problem solving code. While preempitive threads allow rapid prototyping they do not scale well at all. Forking and event driven systems offer much better solutions IMO>

Yes and no. The key to any problem solving endeavor is an accurate definition of the problem; many people suffer from the "hammer syndrome"--using a familiar tool to "solve" (or at least attack) unfamiliar problems. In this case, the key element is a large mass of fairly stable data that has to be manipulated. The constraints are more time related than anything else.

Yes, parallel programming, grid computing, and simultaneous threads running on various nodes is a problem; the solution, as always, is in accurate data modeling. Specifically, fully understanding both the nature and the constraints of the problem before wildly attempting "solutions."

No, "tending network IO" is not a problem. Using either grid computing or a Beowulf configuration with an adequately designed application, the network is almost invisible--for all practical purposes, the calculations and manipulation take place on a single node. That is the purpose of both approaches; it is less a case of "divide and conquer" than "everyone work together."

Does everyone need Beowulf? Not really. We use it because we continually search our database for new ways to solve old problems, and every time we find one, we have to recalculate the entire database to be certain that moving a piece "here" does not negatively impact an output "there."

This is most easily understood by comparing it to the approach of the average, "sophisticated" system seller. App A generates a return of 10% using algoritithms X during the month of May--a 10% profit is claimed. The poor results during the month of June cause an "upgrade"--algorithms X now become algorithms Y, which "would have predicted" a couple of large payoffs, resulting in a 10% profit for the month of June. A "consistent profit of 10% or more" is claimed for the entire period.

Unfortunately, because the algorithms were only tweaked after the fact, very little is understood (or researched, because it is not profitable to do so, unless you are actually betting on your output, rather than marketing it) about the results of using algorithms Y during the month of May. It is almost guaranteed that although algorithms Y "would have been" profitable during June, those same algorithms--if actually used--would have resulted in a substantial loss for the month of May, rather than the "profit" claimed.

Our efforts have been to avoid such fallacious reasoning and "marketing puffery" to arrive at a new set of algorithms that would not only prove more profitable in June, but at least as profitable as the current set of algorithms in all the races analyzed in the entire dataset.

Anyone can come up with a historically based method that showed a profit on paper. To create a method that shows a profit in the real world on races yet to be run is a substantially more difficult endeavor.
Good luck

[GameTheory]

For someone who likes to avoid "marketing puffery" you sure are good at it. You reply to a general statement about grid computing with yet another sermon from the mountaintop of your own ego.

Get over yourself.

[traynor]

GameTheory wrote "get over yourself." An interesting idea, but not really related to my response, which was aimed toward osophy_junkie's comment that <The key to clustering is how you segment the problem being solved. The less time you spend tending network IO the more time the CPU can spend on your problem solving code. While preempitive threads allow rapid prototyping they do not scale well at all. Forking and event driven systems offer much better solutions IMO.>

If you are interested in grid computing, please be a bit more specific in what you ask. If you are clueless about the topic, please don't revert to petty ad hominem attacks to cover that lack of knowledge. Grid computing is a highly debatable topic (regarding its value and usefulness) that is favored by those who believe that if they can only gather a bigger mountain of numbers, it will somehow "mean" something. Unless the basic logic is sound, the numbers generated, and the conclusions based on those numbers, are drivel.

The "flaws" of the system sellers is not in the approach to grid computing, but rather in the defective decision models used. In most cases, an improved decision model would give substantially more advantage to a prospective bettor than the best Beowulf setup he or she could find.

My basic premise still stands; someone pushing a software app that does little more than sixth-grade arithmetic number massaging, along with printouts of numerous "ratings," is essentially pushing worthless junk. The average handicapper could do better with a DRF and a handheld calculator.
Good luck

[traynor]

GameTheory wrote <For someone who likes to avoid "marketing puffery" you sure are good at it. You reply to a general statement about grid computing with yet another sermon from the mountaintop of your own ego.

Get over yourself.>

My apologies. I don't keep track of who said what way back when, or why. I did not realize that you were the "multidimensional space" person.

My comments are not intended to be pontifical. They are intended to communicate ideas in understandable terms, while avoiding jargon and obscure phrasing that obfuscates both content and intent. Topics like neural networks and grid computing may be outside the knowledge set of the average reader. That does not mean that those readers should be excluded with obscure language use intended to create the (in most cases false) perception of expertise. The topics are not complex, and are quite the opposite; grid computing is no more conceptually complex than a number of batteries hooked together.

To include other readers as much as possible, I intentionally try to use non-exclusionary language; specifically, language that does not surreptitiously convey the impression of expertise, unless that expertise is earned, or is valid. It is an intention you might find valuable to try yourself. That is not meant to be insulting, but rather an objective evaluation of your syntactical choices. You seem more intent on gaining the mantle of expert, with little to support that intent, than on communicating or exchanging information or ideas. The problem may be simply in your choice of language; what Ruth Boetcher-Joeres so elegantly referred to as "obfuscatory prose."

If you want to play the role of expert, please do so. I have nothing to gain or lose either way. However, when you get into commentaries about ego and sermons on the mount, it might be useful for you to take a long, hard look in the mirror and ask yourself whether you are referring to me or to yourself.

Bottom line. I don't need the mantle of expert to support my ego. I have the money.
Good luck.

[GameTheory]

Please don't revert to petty ad hominem attacksMy attack was not ad hominem. It was a direct criticism of your persistent rude behavior and condescending attitude. You are a narcissist who cannot see over your own upturned nose to actually read and respond to what is actually being written. Not ad hominem at all. Quite specific.

[traynor]

GameTheory wrote <My attack was not ad hominem. It was a direct criticism of your persistent rude behavior and condescending attitude. You are a narcissist who cannot see over your own upturned nose to actually read and respond to what is actually being written. Not ad hominem at all. Quite specific.>

My mistake. I should have asked you if you knew what ad hominem means. In general, you seem to take yourself way too seriously, and far more seriously than I do. In this world, your opinions are your own, and you are entitled to think whatever you choose to think. I think that amateurs prattling on about their silly theories is incredibly boring, but that, too, is simply a personal opinion.

As for actually reading and actually responding to what is actually written, that implies that what was written is worth a response. In many cases, the comments are observations, not questions. As such, it does not seem reasonable to assume that a response is either desired or necessary. In particular, if I make an observation or venture an opinion, I neither expect nor desire a response. If I ask a question, or someone else asks a question, that is the time to respond--it is part of the process.

For example, there is a large list of opinions about Mikey Pizzola; very few seem to require responses. The postings give voice to opinions, which is the nature of a forum. I have my own opinions about Mikey, which I keep to myself. That, too, is in the nature of forums. Specifically, there is no obligation on anyone's part to respond, nor to express his or her opinion, on this forum or any other true forum.

It is an interesting topic. You may be interested in exploring Suzie Kramarae's muted voice theory. I'm sure it would prove an enlightening, and perhaps beneficial, experience for you.
Good Luck

[GameTheory]

I think your making constant ad hominem attacks against the posters of this forum is rude. That is my opinion, yes. You seem to enjoy psychoanalyzing others. What would you say about a person who feels the need to put a slightly veiled insult or two in practically every sentence they write?

[mainardi]

I'm not too familiar with SVMs, but I do know enough about NNs to be dangerous. Even the simplest design forgets one important factor. A net requires a significant number of events -- races in this case -- to be properly trained. That's fine for smart bombs that scan the terra firma in order to decipher the difference between outhouses and temporary missile launchers while en route to a target. But in horse racing, you have an average of 9 or 10 events per day. While the multitude of factors presented in each race may lead one to believe that a net is a good fit for handicapping, it simply is just that... too simple.

Whereas someone who has PROVEN that it can be done on a regular and consistent basis cetainly may be able to explain their case, my formal (academic and professional) background in AI says there are problems lurking.

Here are just a few issues:
1. Horses can't reason, they only know how to run! While people may be able to influence the outcome in some cases, it's up to the horse to do the real work. Ever heard a losing jockey say that his/her horse didn't like the surface, and then that horse comes back to win next out? Sorry guys, but you can't program that!
2. By the time the net has been trained, the data may already be rendered unusable by any number of outside forces. When the net can predict the effects of a jockey change, what three straight days of rain will do to the "bias-of-the-day", which drop off the claim is a sure thing... THEN the neural net will have done its job.
3. The time and money expended to do everything that a neural net would need to do to have even a remote chance of showing long-term profit isn't worth the investment (just my opinion). Heck, I write commercial handicapping software, and just working out issues with the standard mathematical formulas and heuristics (expert knowledge) absorbs enough of my "free" time. Maintaining a net -- or even designing it to be low-maintenance -- is a nightmare (as I've heard from people who get paid to do that sort of thing).

Just think about this... how many commercial handicapping neural net programs have been introduced in the last five years? Last time that I checked, that number was ZERO. How many have survived since the first one I saw in 1990. My last research didn't turn up much.

My point is that this is so often seen as a complicated game, but between what we get from the raw data (either by using software or done by hand) and what we process in our own personal biological neural nets should be sufficient. The problem is in the race selection and money management areas... that's a much better study for a neural net.

[traynor]

GameTheory:
Is that a rhetorical question (essentially posted for its effect on the poor, hapless reader of this drivel), or one that you expect will be answered? Again, I think you take yourself (and your comments) far too seriously. Lighten up; life is beautiful, the sun is shining, the birds are singing, and another race is going to go off any minute.
Good Luck

[traynor]

mainardi wrote <My point is that this is so often seen as a complicated game, but between what we get from the raw data (either by using software or done by hand) and what we process in our own personal biological neural nets should be sufficient. The problem is in the race selection and money management areas... that's a much better study for a neural net.>

totalmente de acuerdo. No matter now sophisticated the app, it requires thoughtful analysis to convert the data to usable information.
Good Luck