Automating Scratches

[xtb]

Originally Posted by classhandicapper

I don't understand why you went through all this.

I just download the XML file, open it in excel, it gets translated and puts each field into neat columns with headings, and then I import it into my database. It literally takes me a minute. I do it once late morning when the east coast tracks I might play come in and then once later in the day if I am going to play west coast tracks.

Where I've had huge difficulty is with the Timeform API. No matter what I did I couldn't translate the XML in a similar fashion and I'm way too lazy at this point to learn how to do it or write a parser. So I threw in the towel.

I find text files way easier to work with anyway.

Do you still work for DRF/TimeformUS ?

[classhandicapper]

Originally Posted by xtb

Do you still work for DRF/TimeformUS ?

No. I retired last year. I wanted more time to relax and enjoy life and found myself locked down in the middle of global pandemic.

[Jeff P]

Originally Posted by classhandicapper

I don't understand why you went through all this.

I just download the XML file, open it in excel, it gets translated and puts each field into neat columns with headings, and then I import it into my database. It literally takes me a minute. I do it once late morning when the east coast tracks I might play come in and then once later in the day if I am going to play west coast tracks.

Where I've had huge difficulty is with the Timeform API. No matter what I did I couldn't translate the XML in a similar fashion and I'm way too lazy at this point to learn how to do it or write a parser. So I threw in the towel.

I find text files way easier to work with anyway.

I agree that text files are far easier to work with.

However, one of the reasons I initially decided to parse the xml file was that upper management at Equibase assured me they were committed to using xml and that the basic structure of the xml would be permanent.

No. Not permanent forever. But permanent as much as any piece of technology can reasonably be expected to remain intact in this day and age.

There would always be a track node, a node for each race, and inside of each race node the individual scratch nodes for each of the scratched horses in that race, etc.

That was important to me. Before Equibase introduced the scratches and changes xml, I was parsing scratches and changes from the individual .html pages at the Brisnet Supertote site.

It seemed like Brisnet was constantly making html changes and breaking my code. (No bueno.)

A second reason I decided to parse the xml file instead of the individual files or .html pages was that the xml file contains scratches and changes for every track running each day.

That simplified things for me.

All I had to do was write a program to parse one simple xml file over and over each day.

For me this was actually easier than managing navigation events to and parsing events of (say) dozens of individual files or .html pages. (Separate files or .html pages for each of the many tracks running each day.)

Also, back in the beginning, upper management at Equibase told me the reason they were developing the xml file in the first place was to create a single source where not just horseplayers but everybody in the industry could go for scratches and changes constantly updated in something that approaches real time.

Then there's this, which is basically correct:

Originally Posted by headhawg

Jeff may have needed to use it programmatically. He couldn't tell his users to use Excel, save the file, and then import into JCapper. If it was for personal use, sure, use Excel. But coders like to code.

All of that was enough to convince me to invest in the coding hours needed to parse the xml file.


-jp

.

[CBYRacer]

Originally Posted by Jeff P

I agree that text files are far easier to work with.

However, one of the reasons I initially decided to parse the xml file was that upper management at Equibase assured me they were committed to using xml and that the basic structure of the xml would be permanent.

No. Not permanent forever. But permanent as much as any piece of technology can reasonably be expected to remain intact in this day and age.

There would always be a track node, a node for each race, and inside of each race node the individual scratch nodes for each of the scratched horses in that race, etc.

That was important to me. Before Equibase introduced the scratches and changes xml, I was parsing scratches and changes from the individual .html pages at the Brisnet Supertote site.

It seemed like Brisnet was constantly making html changes and breaking my code. (No bueno.)

A second reason I decided to parse the xml file instead of the individual files or .html pages was that the xml file contains scratches and changes for every track running each day.

That simplified things for me.

All I had to do was write a program to parse one simple xml file over and over each day.

For me this was actually easier than managing navigation events to and parsing events of (say) dozens of individual files or .html pages. (Separate files or .html pages for each of the many tracks running each day.)

Also, back in the beginning, upper management at Equibase told me the reason they were developing the xml file in the first place was to create a single source where not just horseplayers but everybody in the industry could go for scratches and changes constantly updated in something that approaches real time.

Then there's this, which is basically correct:


All of that was enough to convince me to invest in the coding hours needed to parse the xml file.


-jp

.

This is really interesting stuff, Jeff. Thanks for sharing!

[ubercapper]

Jeff,



Thanks. As you can imagine this wheel has many cogs, or his octopus has many tentacles.



In any event, Chart Callers do the best they can to keep up with last minute changes, and particularly with track condition changes they may not show up in the XML until up to 10 minutes after they're entered because (I believe, and would have to verify) they are updated when the sync file goes out to all downstream customers. If something (like a change in track condition) occurs while other things are going on (like finishing the footnotes on a chart) that could cause a slight delay as well.



I know it might be pain, but feel free to note any lags in posting track condition information into the XML compared to another source and send me an email (Send me a PM if you don't have my email address) and I'll track it down because if there's anything which can be done I'll look into it.

[Tom]

Originally Posted by classhandicapper

No. I retired last year. I wanted more time to relax and enjoy life and found myself locked down in the middle of global pandemic.

So much for that extra $600 a week!

Good timing!

[classhandicapper]

Originally Posted by Tom

So much for that extra $600 a week!

Good timing!

Yeah, I timed my retirement kind of like some tracks time their races.

[Jeff P]

Gulstream Fri 09-11-2020 -- Bad track condition in the Equibase xml all races:

Right now as I type this, races 1 and 2 at Gulfstream have been completed, and the Equibase xml contains bad track condition for all of today's races.

Track condition displayed on track video is sloppy/sealed.

Races 1,3,5,7,9 are off turf - and track condition for those races is missing from the xml.

Races 2,4,6,8 are on the dirt - and track condition for those races is listed as muddy in the xml.

I wish this were an isolated occurrence, but that is not the case.

Decided to start posting about bad track condition in the xml publicly again in hopes of seeing some improvement.

Reporting these privately to Equibase over the past several weeks without seeing improvement has me feeling like I am helping them sweep this under the rug.


-jp

.

[JerryBoyle]

Originally Posted by Jeff P

Gulstream Fri 09-11-2020 -- Bad track condition in the Equibase xml all races:

Right now as I type this, races 1 and 2 at Gulfstream have been completed, and the Equibase xml contains bad track condition for all of today's races.

Track condition displayed on track video is sloppy/sealed.

Races 1,3,5,7,9 are off turf - and track condition for those races is missing from the xml.

Races 2,4,6,8 are on the dirt - and track condition for those races is listed as muddy in the xml.

I wish this were an isolated occurrence, but that is not the case.

Decided to start posting about bad track condition in the xml publicly again in hopes of seeing some improvement.

Reporting these privately to Equibase over the past several weeks without seeing improvement has me feeling like I am helping them sweep this under the rug.


-jp

.

I wonder if the only solution to this is maintaining one's own database of weather history and up to date weather conditions. I guess you won't have any of the info around how the track is managing the weather such as "sealed".

[Jeff P]

More of the same again today at Gulfstream --

R1 was just completed.

Current dirt track condition is missing from the xml for all of today's off turf races... R1, R4, R7, R9, R11

Imo, if someone at the track can get correct track condition into the track video system: there has to be a way to get that same track condition into the xml.



-jp

.

[Jeff P]

Originally Posted by JerryBoyle

I wonder if the only solution to this is maintaining one's own database of weather history and up to date weather conditions. I guess you won't have any of the info around how the track is managing the weather such as "sealed".

Interesting you should mention that.

The thumbnail below shows estimated residual moisture in the ground at Gulfstream a few minutes before post time for R2.


-jp

.

[JerryBoyle]

Originally Posted by Jeff P

Interesting you should mention that.

The thumbnail below shows estimated residual moisture in the ground at Gulfstream a few minutes before post time for R2.


-jp

.

Interesting. Where do you source your weather data? I downloaded a ton of historical data from NOAA, but I haven't found a good up to the hour source to use on race day (yet).

Also, how does one calculate residual moisture in the ground? Is it a function of pressure, humidity and past rain fall? Do you also somehow have data on the type of soil at each track?

[classhandicapper]

Originally Posted by Jeff P

Interesting you should mention that.

The thumbnail below shows estimated residual moisture in the ground at Gulfstream a few minutes before post time for R2.


-jp

.

That data is VERY interesting to me. I didn't even know it was available. Many years ago when I'd go to Belmont/Aqueduct they would display the cushion depth prominently as you entered so you could see if the track was deeper than usual, if the rail was deeper than the outside etc.. I didn't track it and wasn't sharp enough to exploit it, but I used to try to find a pattern in the dead rails and extreme biases that were more common back then.

Today, the tracks seem to be different. Moisture seems to matter a lot.

There may be patterns to speed and closer biases in the moisture content and how it changes during the course of the day. I believe that's the case, but I have no data to support it. It's just based on some observation.

Of course, it could also help making speed figures. If there was a big change in the moisture content and a race came up suspiciously fast or slow, you'd have some evidence to support breaking it out.

[Tom]

Can moisture date be all that helpful?
Is it uniform throughout the whole track?

I would think it would vary depending on sunlight - say Aqueduct, where the clubhouse turn is used very little, or Belmont, where it is hardly ever used. What about shadows from the clubhouse in the stretch?

[classhandicapper]

Originally Posted by Tom

Can moisture date be all that helpful?
Is it uniform throughout the whole track?

I would think it would vary depending on sunlight - say Aqueduct, where the clubhouse turn is used very little, or Belmont, where it is hardly ever used. What about shadows from the clubhouse in the stretch?

I agree.

All I can say is that I've been at the track where halfway through the card I was convinced there was a speed/closer bias, then the maintenance crew either didn't add water for a few races or added extra water and the track changed. I've even seen it twice on the same card (imo the track was favoring speed, then they allowed the track to dry out and it got more tiring, then they added a ton of water before the feature race and the track was back to speed favoring)

I've never been able to exploit it because I'm not there every day making notes looking for patterns, but if I had good information it might help analyze the results or help make speed figures.