Batch Processor

[Handiman]

This is a bit of an update and a request for some help.

I have a new secondary control window. It resembles Doug's multi-buttoned box. It has three buttons....'Batch Processor'......'Database'...Handifast M-T 6 the first two don't work as of right now. Just the Handifast button.

Handifast will kick out 4 csv files now and have track 5 and 6 still to code. But needed a break.

I have started coding the Batch Processor. So here's the question. What are the parameters that the program should use. Since it will all be automatic and not the result of a race a user just handicapped, not sure what to do. When a user handicaps manually, they either accept default settings, create a custom setting or load a previously created custom setting. But in an automatic process, that step is skipped, well as of right now. Thus the question...what do I do?

Handi

[headhawg]

Here's the way that I was working on it as the object of batch processing (for me) is to pump out a lot of csv files. The users would put the .drf or .mcp files along with the .xrd files into some folder structure, Either all in the same folder (not a great design) or have the files separated out by year and by type (a much better solution). Let the user choose which weight file to use for the batch processing or have the program use some default (like the actual default or a user designated one). I would then let the user choose a range of race dates to process. The program would read the datafiles, match it up to the results files (if found, then process the races), and then output the csv files until finished. Then the users could import the csv files into their Excel program or database to do research like hcap has done.

[Handiman]

I wasn't completely clear. But Head you have brought up something I wasn't even contemplating and that's including results files. The other stuff was I pretty sure of, except for the weights. That is what I was not sure of as I wrote that post.

While trying to go to sleep last night the ole brain wouldn't shut down. And it dawned on me that I could, at the beginning of the batch processing give the user the 3 options they have at the beginning of doing the races manually.

Who produces the xrd result files?

Handi

[mmdorn]

Hi,

Brisnet has the xrd files.

Mike

[raybo]

Although I'm not a DB guy, not even close, IMO, the purpose of a racing database is to allow the user to ask 2 questions:

1. "How has this approach worked in the past under these conditions".

2. "How might this approach work under these conditions".

In order to answer these 2 questions, as well as can be answered with the data samples available, there should be much flexibility designed into the "front-end".(?)

Yes, you must absolutely have the results included, and, either the user must be able to fully adjust the weights of every variable, prior to the processing of the data (during the batch processing process), or he will be required to adjust the weightings in the database, which IMO, defeats at least some of the purpose of having a database integrated into HandiFast.

I know that Hcap suggested that the "what ifs" could be handled by the DB, but, there will be many other "what ifs" users will be wanting to do with the DB, besides experimenting with different weightings, like creating their own "composites", etc., and, IMO, whatever steps can be handled, before the DB portion, the better.

[headhawg]

Originally Posted by raybo

Yes, you must absolutely have the results included, and, either the user must be able to fully adjust the weights of every variable, prior to the processing of the data (during the batch processing process), or he will be required to adjust the weightings in the database, which IMO, defeats at least some of the purpose of having a database integrated into HandiFast.

I know that Hcap suggested that the "what ifs" could be handled by the DB, but, there will be many other "what ifs" users will be wanting to do with the DB, besides experimenting with different weightings, like creating their own "composites", etc., and, IMO, whatever steps can be handled, before the DB portion, the better.

I'm guessing maybe this is what Handi has in mind, but these added features would take Handifast into a whole other realm if it comes to fruition. One of the biggest stumbling blocks is that Liberty Basic (the language Handifast is coded in) does not have any built-in db capabilities. I am not sure how much investigating into adding a db Handi has done but it will be quite the undertaking. So in the interim having batch processing to produce csv files en masse would be a satisfactory solution I would think.

[raybo]

Originally Posted by headhawg

I'm guessing maybe this is what Handi has in mind, but these added features would take Handifast into a whole other realm if it comes to fruition. One of the biggest stumbling blocks is that Liberty Basic (the language Handifast is coded in) does not have any built-in db capabilities. I am not sure how much investigating into adding a db Handi has done but it will be quite the undertaking. So in the interim having batch processing to produce csv files en masse would be a satisfactory solution I would think.

I agree completely, as I said, I'm not a DB guy, but I have tried creating them, and, believe me, it's one pain in the behind.

When Handi mentioned including a DB in HandiFast, I was a little shocked. Having the batch processing feature was really all I/we were looking forward to, and, if that's as far as Handi goes with it, that will be great.

I didn't mean to put any added pressure on Handi, or any of you 3 guys. What has taken place with HandiFast is nothing short of amazing!

[hcap]

Originally Posted by headhawg

Here's the way that I was working on it as the object of batch processing (for me) is to pump out a lot of csv files. The users would put the .drf or .mcp files along with the .xrd files into some folder structure, Either all in the same folder (not a great design) or have the files separated out by year and by type (a much better solution). Let the user choose which weight file to use for the batch processing or have the program use some default (like the actual default or a user designated one). I would then let the user choose a range of race dates to process. The program would read the datafiles, match it up to the results files (if found, then process the races), and then output the csv files until finished. Then the users could import the csv files into their Excel program or database to do research like hcap has done.

Yes. I create 2 folders. One for data files, and 1 for result files. I believe the majority of users use the XRD results files or have access to them. So far what I have done is to import the Handifast csv files into an Excel spreadsheet. And import matching XRD files. I will scratch horse accordingly, but the Handifast ratings -I don't think-will be adjusted as if scratches were done within the Handifast program itself. So yes, you need to give the user the option to import the results and scratches and race cancellations(those races to be skipped and not be added to the database).

Daily play settings could be enhanced by conclusions reached in DB study, and those settings or weights should be available for daily play. From the DB possibly in addition to Handifast itself.

[douglasw32]

Having the batch processing feature was really all I/we were looking forward to, and, if that's as far as Handi goes with it, that will be great.

With so many database's out there, Access, MySQL, Sql Lite, postreq (open source), and a bunch od lesser known 3rd parties.

Simply getting the csv file out would be the best start for most looking for this kind of thing.

BUILDING a DB inside the osftware would put what someone who knows DBing into the hands of Everyone, minus the learning curve, and would be amazing for them.

but if handi is going to go in baby steps, the csv file would be the first place to start and release.

[Handiman]

Handi's head hurts...

Not sure I understand. If xrd files are importable to Excel why would I need to include processing xrd files into a batch file when they could be just downloaded and imported as they are? Am I missing something?

Creating the database in Liberty Basic should be no problem. The problem comes when allowing the user to write queries and being able to process those queries. I almost have to create a whole new language to do that.

Since I know LB fairly well I can write queries in LB to do my own research. But to cover everything you guys might want to do...well I don't know if my brain is that big....

The reason I was thinking of adding database capabilities to Handifast is because I don't know squat about Excel nor do I have Excel. I use Open Office to look at Excel files and do any tiny spreadsheet stuff I want to fool around with. I've never been a database guy before as I have already mentioned. But I love to tinker, so I though why not add DB to Handifast.

I am looking into how to be able to make it so anyone who would like to use handifast for research could do so without having to know how to write LB masks. That may take me quite some time. So I will concentrate on first getting the csv out put for all six track modules. And then finish the batch processor.

Then maybe.....I'll conquer the database mountain.

Handi

[raybo]

Originally Posted by douglasw32

With so many database's out there, Access, MySQL, Sql Lite, postreq (open source), and a bunch od lesser known 3rd parties.

Simply getting the csv file out would be the best start for most looking for this kind of thing.

BUILDING a DB inside the osftware would put what someone who knows DBing into the hands of Everyone, minus the learning curve, and would be amazing for them.

but if handi is going to go in baby steps, the csv file would be the first place to start and release.

I agree!

Having a full featured application that would allow both daily play and historical databasing, without the high price usually involved in such an application, would allow many players to have the same tools, used by some, that would otherwise not have those tools.

I have to admit that part of what caused me to offer my "template" for Excel was in hopes of encouraging someone, knowledgeable in racing database construction and usage, to do the same, create a racing database "template" for a platform, such as Access, that most owners of PCs already have, or could obtain at minimal cost.

[headhawg]

Originally Posted by Handiman

Not sure I understand. If xrd files are importable to Excel why would I need to include processing xrd files into a batch file when they could be just downloaded and imported as they are? Am I missing something?

The results files contain the info about the scratched horses. If you're batch processing past races it becomes meaningless if the scratches aren't included as you would a) either have to do it manually (where we are now) or b) have distorted information in the db.

Originally Posted by Handiman

Creating the database in Liberty Basic should be no problem. The problem comes when allowing the user to write queries and being able to process those queries. I almost have to create a whole new language to do that.

Now I am confused about your intended db feature set so I hope that you will clarify. I probably misunderstood your original post. What I mean is that without the query capability having the db module is not very useful, especially considering the amount of work involved. The good news is that you wouldn't have to create a new language. The purpose of Cheetah, Tsunami, SqlLite, etc is to include the commands to query the database; you just need the correct LB wrapper.

I think the hard part will be having a good database design and implementing the front end. If you make the user type the Select statements only the serious db users will even attempt to use it. My homegrown project will include db capabilities so I've been messing around with it. But for serious db coding LB is not the way to go.

[Handiman]

One thing I am really not happy about Head, is I went on the Liberty basic conforum site and asked the question which would be better to use, sqLite, Tsunami or Cheetah. No one bothered to answer.

So I went into another thread and asked about database programming. Still very little help. I downloaded Cheetah and expected to use it. But then someone finally came on and said why not just use LB. But that was all I could get.

So I had planned to use one of the three and just interface it with Handifast. There is a wrapper available, but I am having trouble getting it. So right now I am concentrating on the csv code to get that finished and looking into the DB. I haven't decided what or which way to go. And I did plan on adding in results file, just wasn't sure which ones. Wanted to get them a cheaply as possible.

Handi

[headhawg]

The forums are usually pretty helpful there, but as I posted previously LB is not really great with DBs so it's no wonder there is not much help being provided. If you look at the number and dates of the posts in that section you'll know what I mean. That's also why I mentioned looking into other languages for db capability as the speed, reliability, and hassle using LB for databases might be more trouble than it's worth.

[hcap]

I brought up scratches because if I scratch in Excel I will not produce the same ratings as if the scratches are done in Handifast.

A relational database may be more than needed-for most users. Right now you are already pulling out some data that is relational from the data files, trainer and jockey ratings. Unless your planning on more factors in the future from the data files, which factors right now should be another table?? Seems good enough as a flat file. No reason a built in database module could not be included in the new Handifast program, but I don't think it has to be very elaborate.

I already have a flat file database set up in Excel. I am more than willing to make it available for Excel users or anyone else. Just have to alter it to accept results and scratches in the new Handifast batch program. Could be used until the database module is done. I rank most factors. A very involved "and" query can be done quickly using all or any factor. For instance "how does the top ranked Fair Odds horse do when ridden by one of the top 5 ranked Jock"?**
On a fast track? On what surface? With a top 3 ranked trainer?

One other point. You may want to give the user a choice of testing with scratches or without.

**Quite well