Batch Processor

[Handiman]

This is a bit of an update and a request for some help.

I have a new secondary control window. It resembles Doug's multi-buttoned box. It has three buttons....'Batch Processor'......'Database'...Handifast M-T 6 the first two don't work as of right now. Just the Handifast button.

Handifast will kick out 4 csv files now and have track 5 and 6 still to code. But needed a break.

I have started coding the Batch Processor. So here's the question. What are the parameters that the program should use. Since it will all be automatic and not the result of a race a user just handicapped, not sure what to do. When a user handicaps manually, they either accept default settings, create a custom setting or load a previously created custom setting. But in an automatic process, that step is skipped, well as of right now. Thus the question...what do I do?

Handi :)

[headhawg]

Here's the way that I was working on it as the object of batch processing (for me) is to pump out a lot of csv files. The users would put the .drf or .mcp files along with the .xrd files into some folder structure, Either all in the same folder (not a great design) or have the files separated out by year and by type (a much better solution). Let the user choose which weight file to use for the batch processing or have the program use some default (like the actual default or a user designated one). I would then let the user choose a range of race dates to process. The program would read the datafiles, match it up to the results files (if found, then process the races), and then output the csv files until finished. Then the users could import the csv files into their Excel program or database to do research like hcap has done.

[Handiman]

I wasn't completely clear. But Head you have brought up something I wasn't even contemplating and that's including results files. The other stuff was I pretty sure of, except for the weights. That is what I was not sure of as I wrote that post.

While trying to go to sleep last night the ole brain wouldn't shut down. And it dawned on me that I could, at the beginning of the batch processing give the user the 3 options they have at the beginning of doing the races manually.

Who produces the xrd result files?

Handi

[mmdorn]

Hi,

Brisnet has the xrd files.

Mike

[raybo]

Although I'm not a DB guy, not even close, IMO, the purpose of a racing database is to allow the user to ask 2 questions:

1. "How has this approach worked in the past under these conditions".

2. "How might this approach work under these conditions".

In order to answer these 2 questions, as well as can be answered with the data samples available, there should be much flexibility designed into the "front-end".(?)

Yes, you must absolutely have the results included, and, either the user must be able to fully adjust the weights of every variable, prior to the processing of the data (during the batch processing process), or he will be required to adjust the weightings in the database, which IMO, defeats at least some of the purpose of having a database integrated into HandiFast.

I know that Hcap suggested that the "what ifs" could be handled by the DB, but, there will be many other "what ifs" users will be wanting to do with the DB, besides experimenting with different weightings, like creating their own "composites", etc., and, IMO, whatever steps can be handled, before the DB portion, the better.

[headhawg]

Yes, you must absolutely have the results included, and, either the user must be able to fully adjust the weights of every variable, prior to the processing of the data (during the batch processing process), or he will be required to adjust the weightings in the database, which IMO, defeats at least some of the purpose of having a database integrated into HandiFast.

I know that Hcap suggested that the "what ifs" could be handled by the DB, but, there will be many other "what ifs" users will be wanting to do with the DB, besides experimenting with different weightings, like creating their own "composites", etc., and, IMO, whatever steps can be handled, before the DB portion, the better.I'm guessing maybe this is what Handi has in mind, but these added features would take Handifast into a whole other realm if it comes to fruition. One of the biggest stumbling blocks is that Liberty Basic (the language Handifast is coded in) does not have any built-in db capabilities. I am not sure how much investigating into adding a db Handi has done but it will be quite the undertaking. So in the interim having batch processing to produce csv files en masse would be a satisfactory solution I would think.

[raybo]

I'm guessing maybe this is what Handi has in mind, but these added features would take Handifast into a whole other realm if it comes to fruition. One of the biggest stumbling blocks is that Liberty Basic (the language Handifast is coded in) does not have any built-in db capabilities. I am not sure how much investigating into adding a db Handi has done but it will be quite the undertaking. So in the interim having batch processing to produce csv files en masse would be a satisfactory solution I would think.

I agree completely, as I said, I'm not a DB guy, but I have tried creating them, and, believe me, it's one pain in the behind.

When Handi mentioned including a DB in HandiFast, I was a little shocked. Having the batch processing feature was really all I/we were looking forward to, and, if that's as far as Handi goes with it, that will be great.

I didn't mean to put any added pressure on Handi, or any of you 3 guys. What has taken place with HandiFast is nothing short of amazing!

[hcap]

Here's the way that I was working on it as the object of batch processing (for me) is to pump out a lot of csv files. The users would put the .drf or .mcp files along with the .xrd files into some folder structure, Either all in the same folder (not a great design) or have the files separated out by year and by type (a much better solution). Let the user choose which weight file to use for the batch processing or have the program use some default (like the actual default or a user designated one). I would then let the user choose a range of race dates to process. The program would read the datafiles, match it up to the results files (if found, then process the races), and then output the csv files until finished. Then the users could import the csv files into their Excel program or database to do research like hcap has done.Yes. I create 2 folders. One for data files, and 1 for result files. I believe the majority of users use the XRD results files or have access to them. So far what I have done is to import the Handifast csv files into an Excel spreadsheet. And import matching XRD files. I will scratch horse accordingly, but the Handifast ratings -I don't think-will be adjusted as if scratches were done within the Handifast program itself. So yes, you need to give the user the option to import the results and scratches and race cancellations(those races to be skipped and not be added to the database).

Daily play settings could be enhanced by conclusions reached in DB study, and those settings or weights should be available for daily play. From the DB possibly in addition to Handifast itself.

[douglasw32]

Having the batch processing feature was really all I/we were looking forward to, and, if that's as far as Handi goes with it, that will be great.

With so many database's out there, Access, MySQL, Sql Lite, postreq (open source), and a bunch od lesser known 3rd parties.

Simply getting the csv file out would be the best start for most looking for this kind of thing.

BUILDING a DB inside the osftware would put what someone who knows DBing into the hands of Everyone, minus the learning curve, and would be amazing for them.

but if handi is going to go in baby steps, the csv file would be the first place to start and release.

[Handiman]

Handi's head hurts...:lol:

Not sure I understand. If xrd files are importable to Excel why would I need to include processing xrd files into a batch file when they could be just downloaded and imported as they are? Am I missing something?

Creating the database in Liberty Basic should be no problem. The problem comes when allowing the user to write queries and being able to process those queries. I almost have to create a whole new language to do that.

Since I know LB fairly well I can write queries in LB to do my own research. But to cover everything you guys might want to do...well I don't know if my brain is that big....:lol:

The reason I was thinking of adding database capabilities to Handifast is because I don't know squat about Excel nor do I have Excel. I use Open Office to look at Excel files and do any tiny spreadsheet stuff I want to fool around with. I've never been a database guy before as I have already mentioned. But I love to tinker, so I though why not add DB to Handifast.

I am looking into how to be able to make it so anyone who would like to use handifast for research could do so without having to know how to write LB masks. That may take me quite some time. So I will concentrate on first getting the csv out put for all six track modules. And then finish the batch processor.

Then maybe.....I'll conquer the database mountain.

Handi:)

[raybo]

With so many database's out there, Access, MySQL, Sql Lite, postreq (open source), and a bunch od lesser known 3rd parties.

Simply getting the csv file out would be the best start for most looking for this kind of thing.

BUILDING a DB inside the osftware would put what someone who knows DBing into the hands of Everyone, minus the learning curve, and would be amazing for them.

but if handi is going to go in baby steps, the csv file would be the first place to start and release.

I agree!

Having a full featured application that would allow both daily play and historical databasing, without the high price usually involved in such an application, would allow many players to have the same tools, used by some, that would otherwise not have those tools.

I have to admit that part of what caused me to offer my "template" for Excel was in hopes of encouraging someone, knowledgeable in racing database construction and usage, to do the same, create a racing database "template" for a platform, such as Access, that most owners of PCs already have, or could obtain at minimal cost.

[headhawg]

Not sure I understand. If xrd files are importable to Excel why would I need to include processing xrd files into a batch file when they could be just downloaded and imported as they are? Am I missing something?The results files contain the info about the scratched horses. If you're batch processing past races it becomes meaningless if the scratches aren't included as you would a) either have to do it manually (where we are now) or b) have distorted information in the db.

Creating the database in Liberty Basic should be no problem. The problem comes when allowing the user to write queries and being able to process those queries. I almost have to create a whole new language to do that.Now I am confused about your intended db feature set so I hope that you will clarify. I probably misunderstood your original post. What I mean is that without the query capability having the db module is not very useful, especially considering the amount of work involved. The good news is that you wouldn't have to create a new language. The purpose of Cheetah, Tsunami, SqlLite, etc is to include the commands to query the database; you just need the correct LB wrapper.

I think the hard part will be having a good database design and implementing the front end. If you make the user type the Select statements only the serious db users will even attempt to use it. My homegrown project will include db capabilities so I've been messing around with it. But for serious db coding LB is not the way to go.

[Handiman]

One thing I am really not happy about Head, is I went on the Liberty basic conforum site and asked the question which would be better to use, sqLite, Tsunami or Cheetah. No one bothered to answer.

So I went into another thread and asked about database programming. Still very little help. I downloaded Cheetah and expected to use it. But then someone finally came on and said why not just use LB. But that was all I could get.

So I had planned to use one of the three and just interface it with Handifast. There is a wrapper available, but I am having trouble getting it. So right now I am concentrating on the csv code to get that finished and looking into the DB. I haven't decided what or which way to go. And I did plan on adding in results file, just wasn't sure which ones. Wanted to get them a cheaply as possible.

Handi :)

[headhawg]

The forums are usually pretty helpful there, but as I posted previously LB is not really great with DBs so it's no wonder there is not much help being provided. If you look at the number and dates of the posts in that section you'll know what I mean. That's also why I mentioned looking into other languages for db capability as the speed, reliability, and hassle using LB for databases might be more trouble than it's worth.

[hcap]

I brought up scratches because if I scratch in Excel I will not produce the same ratings as if the scratches are done in Handifast.

A relational database may be more than needed-for most users. Right now you are already pulling out some data that is relational from the data files, trainer and jockey ratings. Unless your planning on more factors in the future from the data files, which factors right now should be another table?? Seems good enough as a flat file. No reason a built in database module could not be included in the new Handifast program, but I don't think it has to be very elaborate.

I already have a flat file database set up in Excel. I am more than willing to make it available for Excel users or anyone else. Just have to alter it to accept results and scratches in the new Handifast batch program. Could be used until the database module is done. I rank most factors. A very involved "and" query can be done quickly using all or any factor. For instance "how does the top ranked Fair Odds horse do when ridden by one of the top 5 ranked Jock"?**
On a fast track? On what surface? With a top 3 ranked trainer?

One other point. You may want to give the user a choice of testing with scratches or without.

**Quite well

[lsosa54]

Is there any reason not to use the free BRIS .RES results files which come zipped? Seems like a simple text file.

[raybo]

Is there any reason not to use the free BRIS .RES results files which come zipped? Seems like a simple text file.

I think that would be akin to trying to parse a PDF file. Don't believe it could be done accurately. That's the reason that the comma-delimited results/charts files were created, so "programmers" could manipulate the data because the data is always in known field locations.

[Handiman]

I have looked at the .xrd files and it's a matter of processing them just like the mcp files. Hate the .25 per race card. When building a database I'm sure it can get a bit expensive. I get the mcp files for $59 a month.

I will look around a bit for some others.

Handi:)

[Handiman]

Before I go to bed and before I move any further with the batch processor, I have a quick question.

Is this what you guys have in mind as the way the batch processor should work.

Open processor.....select a group of files to process. Box opens and you select default settings...create custom setting or load a setting. Then the processor processes the xrd file and then processes the files all the while checking the corresponding xrd file for scratches and then finish processing file and produce a csv file for each race card. Do you also want a csv file of the xrd file?

Thanks
Handi:)

[headhawg]

I hadn't thought much about the processing of the xrd file. I imagine that having the ITM horses and the prices would be essential as well as the scratches -- not so much to know which horses were scratched but so that they don't get processed.

There's no need to create a csv for the xrd as it already is in comma-delimited format.

[Handiman]

Went to process my first file tonight with the new batch processor, but ran into a bug concerning a file. So I sent HeadHawg a message and ask for some help. If he can straighten this bug out for me, or help me resolve the issue then we may have a kick ass batch processor that will create csv files on all the files you want to run.

I'm pretty excited about this...assuming we can find the bug and squash it.

Handi:)

[Vinnie]

Handi:

"YOU ARE THE MAN"! I can't believe that you pretty much have completed the Batch Processing function for Handifast already.. That is Outrageous. :)

Have a great day and a super weekend.

[Handiman]

UPDATE!

Where's an Orkin man when you need him. I had to go to Head again for some pest control.

But this is where we stand. I finished the Batch processor and it works beautifully. BUT....it will only do one race card. It can't find the second file to do. So I sicked the HH Bloodhound on it, and I am hoping he will be able to look at it tonight and let me know he's found out why it will pull one file out and process it, but can't go back to the same folder and pull another racecard to process.

Later I will come back here and lay out just how it's going to all work.

Handi:)

[Handiman]

Fix one bug and it gives birth to several more. This batch processor is a bitch to program! :bang:

back Later.

Handi:mad:

[headhawg]

This batch processor is a bitch to program!Yes, yes it is. I am giving Handi a ..um..hand with this, and I am hopeful that I can Orkin most/all of the bugs by tomorrow. The linemaker puts the odds of that happening at about 9-5. ;)

[Handiman]

HeadHawg is worth his weight in Bacon! Thanks Head for the help man...I don't know what I'd do without you.

So here is the latest. I just used the batch processor and processed 5 tracks in about 2 minutes. But there is a small hold up, which I am talking to Head about. And that is right now, as the result of testing purposes I have the program ask the user if they want to save the csv file every time a card is processed.

I am leaning toward eliminating that feature or bother as I call it and have the program just list the cards processed as each one is done. But that is going to take a bit more time.

So that is where the BP stands at present.

Handi:)

[headhawg]

UPDATE: I will try to do more with the code this weekend to at least make it close to usable in a "beta" mode. I can say that this is very close to being done.

I'd rather be worth my weight in gold, but bacon is fine as long as it's the good stuff. Oink. :D

[fast4522]

To cut 5 or 6 cards up that fast into CSV files really does lend itself to the guy who might be a Excel guy headed out the door for a tournament. Your ability to address CSV's just because someone asked is quite awesome indeed.