Database Discussion for Handifast

[Handiman]

Looking for some input. I'm working on designing the database. What I want out of the database is to be able to check setups and see what kind of results different ones would produce. Since I am not a database guru I'm sort of befuddled.

It seems to me that every time you want to check a certain setup the entire base of races have to be run using that particular set of numbers. new setup to test then entire base of races have to be run again. Is there an easier way?

Handi:)

[guckers]

Looking for some input. I'm working on designing the database. What I want out of the database is to be able to check setups and see what kind of results different ones would produce. Since I am not a database guru I'm sort of befuddled.

It seems to me that every time you want to check a certain setup the entire base of races have to be run using that particular set of numbers. new setup to test then entire base of races have to be run again. Is there an easier way?

Handi:)

Deciding how you are going to build your database is a lot easier if you have a general idea of what you want to achieve and or test for performance. Spend more time designing than building as changes to your database further in to the development cycles will kill your motivation and make it more difficult to use (measure twice cut once mentality).

Here is a place to start.

http://en.wikipedia.org/wiki/Entity-relationship_model

[Dave Schwartz]

I know you are going for simple. Maybe consider something like this:

http://www.horsestreet.com/BBSImages/Filter01.png

http://www.horsestreet.com/BBSImages/Filter02.png

http://www.horsestreet.com/BBSImages/Filter03.png

http://www.horsestreet.com/BBSImages/Filter04.png

[Handiman]

Here's what I'm thinking. Since Handifast is set up to rank each category and top 4 horses get points in relation to their rank, building a data base would cause all races to be run and each horse would be given rank number in each category. Then user can assign points to the different categories and run query returning results based off of the setup being tested.

Sound about right?

Handi:)

[headhawg]

Maybe I am misunderstanding but in order to find out if the point setup is "optimal", wouldn't you have to run all possible combinations of point setups against the actual results? Basically this is backfitting, but the program could be designed so that the user could split the data -- maybe 75/25 or whatever is statistically best -- and then run all of the combos on the 75% and then see if the settings "go forward" by testing it on the remaining 25%. That way the point model wasn't biased by the results of the 25% as that data was not included.

[Handiman]

Isn't the use of Databases in general back fitting. Finding out what has been successful in the past? Looking for the optimal settings would require doing just what you mentioned Head, but that would be 20 raised to the 10th power iterations I think.

What I was thinking about is everyone has some idea of what factors are important or ones they like to include. Bu doing as I said, they could test out their factors and different point settings for those factors until they found the best setups for their particular factor set.

Anyway that is what I had in my head. But as I said I am looking for as much input as I can get before I start the design and then the coding.

Handi:)

[TrifectaMike]

Maybe I am misunderstanding but in order to find out if the point setup is "optimal", wouldn't you have to run all possible combinations of point setups against the actual results? Basically this is backfitting, but the program could be designed so that the user could split the data -- maybe 75/25 or whatever is statistically best -- and then run all of the combos on the 75% and then see if the settings "go forward" by testing it on the remaining 25%. That way the point model wasn't biased by the results of the 25% as that data was not included.

A nice way to test is to split the data in half. Fit using the first half and test on the second half. Then fit on the second half and test on the first half. Compare results from both and draw conclusions.

Mike (Dr Beav)

[headhawg]

Looking for the optimal settings would require doing just what you mentioned Head, but that would be 20 raised to the 10th power iterations I think. What's 10 trillion iterations among friends, Handi. :D

[JustCoolGene]

User places all available BRIS .drf or Jcapper .jcp and comma delimited results .xrd files for same year into one folder.
__________________________________________________

The user pushes ONE BUTTON and Handifast imports the first .drf\.jcp data file along with the corresponding .xrd results file. Handifast loads data for first race and removes scratches and applies any surface changes discovered in the results file. Handifast then generates all twenty factor rankings and creates a "RACE CATEGORY" key for each horse that describes the Track, Class, Distance, Surface, and age (young or old). Handifast then saves the rankings, race category key, and finish position for each horse to a "RANKED" table.

Handifast processes all races for the card and grabs the next data file in the year folder. Repeat the same procedure until all files for the year are processed. If you had all race cards for the year, you would end up with a "RANKED" table with approximately 40,000 horses.

Handifast then processes the "RANKED" file and builds impact values values for top four rankings for each of the twenty factors in each category. Save these results to a table named "IMPVALUE"

Handifast then looks at each race category in the impvalue file and selects only the top performing factors based on impact values. Handifast then creates a "BESTOF20" results file containing only the best performing factors in each race category. All steps above could be automated and the user ends up with optimized category factors with the push of ONE BUTTON..
__________________________________________________

Now we are ready to play today's races. Place todays .drf\.jcp files into a folder called "today" and push ONE BUTTON. Handifast processes all race cards for today and creates an oddsline for each horse based only on the best performing factors for its race category.

The 40,000 horse RANKED table, the IMPVALUE table, the BESTOF20, and TODAY'S files should be exported as a "comma delimited" files so that database or spreadsheet uses can easily access all the data.

Handifast could automatically print a one page report for each race card containing only minimum horse data and optimized odds-line.

Gene

[headhawg]

Um...Handifast is a free program. There might be some stuff in your post that Handi might use. If you want custom programming I am sure that can be arranged for a fee.

[Dave Schwartz]

JustCool,

What you have described is a very watered-down version of what my software does. I assume that most other commercial products function in a similar manner.

Not quite as automatic, because when you are dealing with several thousand factors for each horse, you've got to choose the factors you want tables for (if that is your chosen method of handicapping).


Regards,
Dave Schwartz
PS: we don't use BRIS files.

[Handiman]

You guys have to remember I'm not a rocket scientist. I had one science class in school and that was Biology. And I spent the whole time looking down the blouse of my beautiful lab partner, so didn't develop the science part of my brain much.

While 'just cool' has some cool stuff laid out there, I am a simple country boy that is thinking that I want to develop a way to ferret out the best setups I can find, without having to have my computer running continuously for the next 3 or 4 years working through the 10 trillion iterations.

Again, I'm between 'Dumb as mud' and 'Smart as a cricket' so I do have coding boundaries.

Handi:)

[JustCoolGene]

You guys have to remember I'm not a rocket scientist. I had one science class in school and that was Biology. And I spent the whole time looking down the blouse of my beautiful lab partner, so didn't develop the science part of my brain much.

While 'just cool' has some cool stuff laid out there, I am a simple country boy that is thinking that I want to develop a way to ferret out the best setups I can find, without having to have my computer running continuously for the next 3 or 4 years working through the 10 trillion iterations.

Again, I'm between 'Dumb as mud' and 'Smart as a cricket' so I do have coding boundaries.

Handi:)

Handi,

Nobody 'Dumb as mud' and 'Smart as a cricket' could ever create Handifast. You enlightened and influenced many on this board with your many contributions.

Everything I described would only take one night of processing on a 3.0Ghz processor and fast hard drive. Processing thirty race cards on a Saturday in June would only take maybe 15 minutes. But that's not the point. If you have coding boundaries, maybe you should work on automating the processing of race cards and building an export table containing the rankings, values, and selections with the push of only ONE BUTTON. Handifast users could then export the table as a CSV file. Database and spreadsheet users would then have easy access to Handifast data. Users could "ferret out the best setups" and re-type into Handifast the newly found optimized setups for different distances and surfaces .

During one of Dave Schwartz HSH Insider's Club Shows, he mentioned that this year he was going to release a stand-alone program which would allow users to import their own custom factors or output from other programs such as Handifast. Users would then have the ability to create consensus type of selections uses no HDW data or any other data source. He said the new software had some very cool features. This new software would be an exciting option for all users of horse racing software. Dave is always thinking of new ways to help horse-players, he is a great asset to this "game of ours".

Gene

[Light]

A simple solution would be to make the output of the program exportable to either Excel or Access or both. My home made program runs in Excel and I have some small macros I made that splices specific info from the program's output into an Excel database.There is more elbow grease involved as I manually enter results in the database. But I only keep serious tabs on 2 tracks at a time so it's not that labor intensive.May be an intermediate solution for you, but for me it fills the needs of a query.

[Handiman]

I started coding the database part of Handifast and as usual I'm doing this ass backward.

I don't know what I'm doing.
I don't know how it's going to work.
I don't know when I'll be done.

So stay tuned I guess. I do have a general idea, but as the coding progresses I'm expecting it too all fall in line.

Handi:)

[Handiman]

Coding Update: I have code done that will allow program to check C:\2012 and match up data files with .xrd results files and then download data and get ready for next step....which would be handicapping to get each horses ranking in each factor after checking for scratches.

So some progress.....yea!

Handi:)

[Handiman]

Holy Crap Elvis. I just built my first Database with code I wrote. A bit excited right now. But just realized I now have to write code to access the database and see if I have put the right info in it. :lol: Forgot about this part. Oh well.

Handi:)

[headhawg]

What engine did you decide to use? Cheetah? Sqllite?

[Handiman]

Neither. I have written it in Liberty basic using random access files with relational tables.

This is all new to me, so it's a bit rough going. But I didn't know how to interface with cheetah or sqlite.

Handi:)

[Handiman]

Looks like I have the database debugged. At least hope so. I am now starting to code the user interface. Not sure how long that is going to take. We'll see.



Handi:)

[headhawg]

Good stuff, Handi. You've inspired me to write code again so I am working on some things, little by little: a new interface, autoscratch, batch processing, and...gulp...a database feature. I really would like to work on a faster data file load routine as well.

I am looking to take HHX into a slightly different direction than Handifast. If things work out I'll see if I can make it available here.

Hopefully things with your database function are going well.

[Handiman]

Head..... Can't wait to see what you come up with soon. Where can I get the latest version of HHX?

Handi:)

[headhawg]

The latest version is still v2.1 from a long time ago. About the only thing I did after that was include a couple of batch files that would load/save different weight configurations, a very primitive version of what you have included in Handifast now. I'm pretty sure that you have that version. It may still be available from Doug's site as well, I'm not sure. If not, just let me know.

When I code something new I'll send it to you. We are looking at at least a month or two. I am just playing around with a new interface now. GUIs are not my strength.

[Handiman]

No problem buddy. Whenever you get something fine. I know how hard it is to get stuff knocked out.

Handi:)

[Handiman]

Update: I have Aqueduct since the first of the year in database. I have the user interface coded and have run a few queries. I found a route dirt that showed nice ROI and by the time the report came up, I had forgotten what numbers I had put in....so I have added text boxes for each factor that displays the number ratings as you put them in, so if you find a setting that shows positive ROI you will have the numbers available to write down.

Next step is to add more race cards to database and make sure that it all works. Then I will have to code the 'Append" ability. After that and some testing I think the program in it's latest incarnation should be ready to go.

Handi:)

[Vinnie]

Handi:

"You, HH and Doug totally Rock"!! Way to push the edge of the envelope.. :)

Have a super day today. I only wish you continued excellent health and wellness sir.

[Handiman]

Thanks Vinnie. This is turning out to be quite a major challenge.

Handi:)

[Vinnie]

Honestly Handi, with what you have shown in how you program in developing Handifast and variations of other programs you have already shared with us to date, I believe that there are very few things that you cannot accomplish when you put your mind to it. :)

I only wish that I could do what you guys do. Thank you for all of your efforts.

Vinnie

[Handiman]

Ran into a major snag and so having to do some clean up and rethinking before moving on......Hopefully back on track soon. This is quite a daunting task.

Handi:)

[headhawg]

It is indeed Handi. I am doing a (nearly) total rewrite of HHX and even the little things seem to be taking forever for me to code. Far down on the coding list is db capabilities but that might turn out to be a totally different program from Handifast HHX.

I've looked at some of the db things available for LB and I'm trying to decide which one to commit to. There's the previously mentioned SQLlite (but there is only an LB wrapper for an older version), and the Cheetah DB (non-SQL based). I've also discovered a B-tree indexing dll that might work with LB which mainly adds a fast indexing system to whatever sequential/random access files exist in the application. Not relational, but for what I might want to do a relational db may not be necessary.

Continued good luck with your foray into the db realm. Knowing your persistence I am sure that you will get it working shortly.