Handicapper's Data Warehouse

[ceejay]

I'm just curious because all software is only as good as the input data (you know, garbage in-garbage out), do users think that HDW data is better or worse than other data sources such as TSN/BRIS, DRF, ...?

If HDW is better, why? Is there better data QC @ HDW than the others?

[Dave Schwartz]

HDW is squeaky clean data. Not just good but ALMOST perfect.

I see an error maybe once or twice per year.

The others I have used are just not as good.


Regards,
Dave Schwartz

[wes]

I will agree with Dave. The best data on the market.

wes

[sq764]

I use Trackmaster data files for Val2 and have zero problems.

[VetScratch]

Dave,

Thanks for your objective and unbiased opinion. :D :D :D :D

Maybe Gary Hall (All-Ways) will offer his opinion. :D :D :D :D

[Dave Schwartz]

VS,

If you wish to take the position that everything I say comes from a biased position, you may certainly do so. But if you check most of my posts I think you will find that historically I am a horse player and PA member first and a vendor second.

Of course, some people might want to consider that I simply have some degree of experience with cleaning up download data myself.


Dave Schwartz

[kitts]

I'll jump in here. I was one of HDW's first employees in 1995. Equibase used to refer to us as "4 men in a garage." This was back in the days when DRF was king and Equibase was getting into the data business and basically sent us the data the tracks sent them. HDW has a wizard, Ron Tiller, who has extremely high standards for data. He instituted very thorough error-trapping procedures. These were so efficient that Equibase incorporated some of them into their own database. I seriously doubt that the low priced data providers have the resources to commit to the cleaning and recleaniing of data. However, as someone once said, there is value in "good enough numbers."
But, only HDW numbers for me.

[VetScratch]

Kitts & Dave,

Testimonials and opinions are fine... but the fact is that content and accuracy are the two important dimensions of merit for data.

In both cases, content and accuracy, data must be judged against specifications.
(1) HDW does not even publish specifications for files or data elements on its web site.
(2) BRIS publishes really poor specifications for files and data elements... you couldn't get a passing grade in a community college IT project with the stuff that BRIS publishes (and I almost said high school, but BRIS deserves some credit compared to zero credit for HDW in this respect).

Until professional specifications provide an objective frame of reference, it is pointless to argue the comparative merit of data content and accuracy.

[sjk]

I get my data direct from the equibase charts. I am interested in the sorts of errors that others find in this data that I need to be on the lookout for. Here are a few that I occasionally see: a). 6's that should be 8's or 1's that should be 4's in the fractions; b). beaten lengths of 21 that should be 2 or 1; c). races run at the wrong distance (1 1/16 miles instead of 1 mile 70 yds).

I see many fewer of these than I did a few years ago. Either the quality of the charts has increased or I am slipping.

What else should I be catching?

[Tom]

Originally posted by VetScratch
Kitts & Dave,

Testimonials and opinions are fine... but the fact is that content and accuracy are the two important dimensions of merit for data.

In both cases, content and accuracy, data must be judged against specifications.
(1) HDW does not even publish specifications for files or data elements on its web site.
(2) BRIS publishes really poor specifications for files and data elements... you couldn't get a passing grade in a community college IT project with the stuff that BRIS publishes (and I almost said high school, but BRIS deserves some credit compared to zero credit for HDW in this respect).

Until professional specifications provide an objective frame of reference, it is pointless to argue the comparative merit of data content and accuracy.


What in the world are you talking about?:confused:

[VetScratch]

Tom,

See the first post in this thread.

This is a software/systems issue.

[Dave Schwartz]

VS,

Well, nice to see you can still shift gears so easily.

I believe this thread was about accuracy. That is what I was addressing.


Dave Schwartz

[VetScratch]

Dave,

Again, see the first post in this thread.

"Software is only as good as the input data..." and "...better or worse..." clearly encompass consideration of content and accuracy.

Do you actually write much software?

[Dave Schwartz]

VS,

Thanks for asking, but I don't wish to play these word games with you.


Dave Schwartz

[ceejay]

Originally posted by Dave Schwartz
I believe this thread was about accuracy. That is what I was addressing.


That was my intent. I accept your comments (not that I don't believe you) with full knowledge that HDW is a business partner of yours.

[Tom]

Originally posted by VetScratch
Tom,

See the first post in this thread.

This is a software/systems issue.

I ask again, what in the world are you talking about?

[Tom]

Originally posted by ceejay
That was my intent. I accept your comments (not that I don't believe you) with full knowledge that HDW is a business partner of yours.

HDW has quality control in place - they check out the data, correct errors, make sure it is as good as they can get it. They do not rush to get it out several days in advance -they would rather do it right. Check out other data sources, ie DRF. They pride themselves in getting ot out as quick as they can, accuracy be damed. Cehck out recent threads in other area here about wrong data, missing data, etc. HDW ius a professional outfit. DRF, is, wel, DRF (Data Really Fast????? <G>)

[Dave Schwartz]

CJ,

Thank you.

If HDW had data problems (and I was still doing business with them) I would simply have been quiet about it.


Dave

[Gold Bay]

I think the real variable lies with the race callers.Litfin wrote about this in Expert Handicappers under pitfals of pace.

[sq764]

Well Dave, I don't know you at all, but I was told you were a nice guy, but very eccentric..

Whoever told me that was very 'accurate'..

[Dave Schwartz]

SQ,

The dictionary says that means a person with an odd personality.

If that is what you meant, why do you say that?

If that is not what you meant, perhaps you would elaborate?

Dave

[keilan]

I always thought of "eccentric" as a term reserved for the wealthy that on occasion tended to act a little quirky at times. However the poor and downtrodden are categorized as crazy when they behave slightly left or right of center. But IMHO you would have to be crazy to engage this fool again anytime soon.


P.S.

Dave are you really wealthy? ;)

[Dave Schwartz]

Keilan,

Actually, yes. (But I am a little short of money.)

Dave

[keilan]

Ah yes, a man with a good heart is very rich indeed!

[VetScratch]

Tom,
Originally posted by Tom
I ask again, what in the world are you talking about? Customers are asked by HDW, BRIS, and other data sellers to purchase data products that lack professional/adequate specification.

HDW does not publish data specifications, and BRIS specifications are woefully superficial.

Customers spend $1200 to $5000+ per year for data.

Does it make sense to spend that much for undocumented or shabbily documented data products.

All the major handicapping programs require essentially the same data to compete in the marketplace.

Does it make sense to spend $-Thousands for data that shackles you to a specific third-party brand of software?

Artificially contrived redundancy serves the purposes the software vendors, but aren't we foolish to fall for this business model?

When you pay $-thousands for data, shouldn't it be professionally documented and readily portable?

Every accomplished programmer who has cracked several data downloads could bear witness to the fact that commerical software products typically do a poor job of reporting data anomolies to users.

I routinely find problems in files that are used by commercial programs which fail to report any deficiencies to users.

When I find time during the coming week, I will illustrate my point with specific examples that everyone (including non-programmers) can understand.

[JustMissed]

Got in on the discussion late, but wouldn't it be to the software vendors advantage to have the capacity to use any data vendors downloads?

You know when you get ready to download your data you just click on:
Bris
TSN
HDW
or whatever.

What's the big deal, the data all comes from the same place.

Think how silly it would be for Ford cars to only use Shell gas or Toyota's to only use Texaco.

The software could have(should have) a filter to detect bad data just like a car engine has a filter to detect bad fuel.

All gasolines come out of the same hole just like all horseracing data comes out of the same race caller's lips.

JustMissed
:)

[VetScratch]

JustMissed,
Got in on the discussion late, but wouldn't it be to the software vendors advantage to have the capacity to use any data vendors downloads?Good points... but we can also look at advantages to consumers who pay the bills.

Another way to look at your automobile analogy is: why is essentially the same gasoline delivered through different hoses that have nozzles which fit only specific brands of cars? Why pull Chevys up to one hose and Fords up to a different hose, when all the gas comes from the same tank?

Of course, since data is not burned, one fill-up could drive several handicapping programs... sort of like consensus handicapping.

Like your analogy, my analogy may be criticized (by non-consumers) as oversimplification... but most analogies are meant to illustrate a valid concept via simplification.

As I posted earlier, I don't have time right now, but I want to post more on the subject of data content and accuracy so that non-programmers can easily understand these issues from a consumer's perspective.

[GARY Z]

Originally posted by Dave Schwartz
HDW is squeaky clean data. Not just good but ALMOST perfect.

I see an error maybe once or twice per year.

The others I have used are just not as good.


Regards,
Dave Schwartz

[Tom]

VS,

I am not clear by what you refer to as specifications-do you mean the structure of the files, what is in the fields?

>Customers are asked by HDW, BRIS, and other data sellers to purchase data products that lack professional/adequate specification. HDW does not publish data specifications, and BRIS specifications are woefully superficial. <

My HDW files are precisely and cleary documented-every field in my export is explained.

>Customers spend $1200 to $5000+ per year for data. Does it make sense to spend that much for undocumented or shabbily documented data products.<

I pay $1400+ a year on the data. I make much more than that in profit using the
data. Essentially, the data is free to me.

>All the major handicapping programs require essentially the same data to compete in the marketplace. Does it make sense to spend $-Thousands for data that shackles you to a specific third-party brand of software? <

Not so. HDW generally, and HTR specifically, provide me with data not found in any other source. Crammer figures are not available from BRIS, TSN, DRF, EB, nobody.
The data is usable in both HTR and much of it is available for export into a database for whatever other use I see fit, short of selling it <G>

>Artificially contrived redundancy serves the purposes the software vendors, but aren't we foolish to fall for this business model?
When you pay $-thousands for data, shouldn't it be professionally documented and readily portable? Every accomplished programmer who has cracked several data downloads could bear witness to the fact that commerical software products typically do a poor job of reporting data anomolies to users.
I routinely find problems in files that are used by commercial programs which fail to report any deficiencies to users.<

HDW/HTR inspects and fixes the data before I get it. I trust them and take their word for it. Why? Because I have a way to test the RESULTS-HTR has a terster function built in, and I have Access to check on my own. I am only interested in results, and HTR gives me the way to monitor them.

I posted Saturday that I had the ability to find a new track, learn all about it, and start betting it in 30 minutes. Aqu was closed, so I went to Calder. In the first race, there were only 4 contenders and only two of those meet the energy demands of the track. The favorites and the winner, which I had, at 19-1. If that is being tied to 3rd party software, tighten those knots! I only looke din the program for what was winning and then a quick loof into Access to see specifically what that type of race was doing lately.

[Tom]

Personally, I want Jim Crammer to make a lot of money. I want Ron Tiller to be rolling the dough. I want Ken Masa dining on cavier and champagne every night.
Because as long as these guys are happy, they are taking me along for the ride. If they aren't making a lot of money selling their products, the might just stop doing it and keeping them to themselves.

[keilan]

[In the first race, there were only 4 contenders and only two of those meet the energy demands of the track. The favorites and the winner, which I had, at 19-1.]

Tom not to redboard because I gave Suff this horse in the WR, but that was a classic of E%. Some of field is still trying to find the wire.

[sq764]

Dave, I have always thought of eccentric to mean not an odd personality, but a unique one..

Unique isn't bad, is it?

[Jaguar]

Tom, nice hit. I gotta take my hat off to anyone that can make a hit like that!

All The Best,

Jaguar

[VetScratch]

Tom,

Your posts were very good... but you described contentment with a vendor-oriented business model. Nothing that you like needs to be sacrificed in a consumer-oriented business model, while many additional benefits would be gained.

Have you considered what would make you happier than you are?

As the mare said to the farm manager on the way out of the breeding shed, "The teasing was great, and Strongheart Dancer gave me good service, but I can't say that I was pleased by that lip shank contraption."

[JustRalph]

Originally posted by VetScratch
As the mare said to the farm manager on the way out of the breeding shed, "The teasing was great, and Strongheart Dancer gave me good service, but I can't say that I was pleased by that lip shank contraption."

I love it when she talks dirty!!!

[VetScratch]

Ralph,

Motorcycle cops and bull riders... you can't fantasize without them, but they sure aren't easy to tame! However, the last bull rider that tried to throw a lip shank on me is now the soprano in a barbershop quartet! :D :D

[JustRalph]

Originally posted by VetScratch
Ralph,
Motorcycle cops and bull riders... you can't fantasize without them, but they sure aren't easy to tame! However, the last bull rider that tried to throw a lip shank on me is now the soprano in a barbershop quartet! :D :D

I have some old police uniforms and a black and white Harley????

Now if I could just find a little flashing blue light!

[RonTiller]

OK, here I go, doing a swan dive into the frying pan (pull me out if I get too brown Dave)...

1. The original question was "...do users think that HDW data is better or worse than other data sources such as TSN/BRIS, DRF? If HDW is better, why?" I am obviously not an objective party, so I'll stay out of the specifics. This is a valid, if somewhat hazy question. What is "better"? How MUCH better (or worse) is one than another. How much DIFFERENCE does it really make in the program or database that uses the data?

2. This morphed into a) Dave's "HDW is squeaky clean data" and b) Kitts' homage to "...very thorough error-trapping procedures..." Now I like endorsements as much as the next guy, but the original set of questions is still unanswered for most people. What does "squeaky clean" mean, how "unsqueaky clean" is other data, and what are the practical consequences for downloaders, if any? Same with "...very thorough error-trapping procedures..." What errors, how pervasive, what consequences?

3. Next level: Vetscratch's "...content and accuracy are the two important dimensions of merit for data." Well, "better", "clean" and "error-trapped" are replaced by "content and accuracy", and publicly available file specs are apparently the essential key to evaluating THAT.

a. I'm unclear on "accuracy" here. Fidelity to reality? No vendor can certify that a horse listed as 5 1/4 beaten lengths at the 3rd call was REALLY 5 1/4 lengths back. Nor can any of us certify that a given horse was REALLY foaled in Kentucky, has been gelded or is owned by [Fill in the Blank]. Nor will I swear to certitude that a horse REALLY ran 5f handily in 59.34 seconds. Now I'm not saying that customers don't find errors in raw data; they do. PaceAdvatage members probably find a disproportionate share of them. But 99.99% of all the data elements are just not checkable by vendors or customers in any reasonable sense or any reasonable time frame; that's why we have Equibase.

b. But maybe the vendors screw up the raw data from Equibase or put data in the wrong place in their own files (in HDW's case, we have the perfect cover, because only the software developers have the file spec). OK, you have to take it on faith that I am not a complete boob. Trust me, Dave did NOT take it on faith when he started using HDW data and neither did Ken Massa or Gordon Pine or Bob Purdy or Charlie Judge or (especially!) John Rancont, Michael Perry or Bill Burns or or anybody else. I would have been extremely disappointed had they not assumed the worst.

c. If not raw data, how about value added data, like pedigree, trainer or horse statistics? Maybe THAT is not accurate. Well, there's probably not much I can say to sway any votes here. While we all make errors, internal checking, as well as following up on reported problems (both real and imaginary) yields a degree of confidence for me, the vendors (who are by far the toughest critics) and ultimately, through using the programs, the customers. An HTR user doesn't need a published file spec to see whether there is a problem with a horse's at- the- track statistics: if it is obvious (last 3 starts at crc yet 0 starts at the track, how does the file spec that the developer uses to translate data from the file to the program help at all; if it is NOT obvious, (no recent races at crc but 47 races ago 1 race at crc yet 0 starts at the track) how could a published file spec used by the software developer help?

d. If not relatively objective value added data, how about the more subjective data, like speed numbers, pace numbers, class numbers, race ratings, power numbers, running style designations, etc. Well, here "accuracy" still has some play, as one can give a horse a boneheadedly wrong speed rating (6 out of 125 for the Derby winner). Also, they might be calculated incorrectly by OUR OWN standards (i.e. a programming error in making a race rating - once made accurate by OUR OWN standards, you can still create threads on paceadvantage.com criticizing our idiotic race rating as being "inaccurate"). Again, a published file spec would be of no value here. These numbers are all displayed wherever and in whatever manner the software developer wants them.

e. Ultimately, since HDW does NOT specifically make multipurpose, generic comma delimited files for consumption by individuals who want, well, multipurpose, generic comma delimited files that they can use however they see fit (many companies ALREADY do that and they have 99% of the customers), the "accuracy" issue, as envisaged by Vetscratch must be dealt with from the context of the program using and displaying the data. So I propose that Dave provide the source code for all his programs, since programmers can screw up perfectly good HDW data. Plus, how do we know Dave is ACCURATELY pulling down the real best of last 3 speed ratings when he says he is. Some of his calculations are hideously complicated. How can I be assured THOSE are accurate? Dave, you get no points for even trying here.

4. From here, the thread morphs into another round of debate concerning HDW's business model (which is vastly different from BRIS's). Sigh... Since I have already fully explained it in other threads, I won't repeat it. If somebody failed to get it then, they won't get it now. If they did get it but still believe HDW is [Fill in your favorite negative phrase, deprecatory remark or outright insult here], then alas, so be it. Most people will not be customers of HDW; for those that are, thank you. Regardless, I'll still be here all day on Thanksgiving, this weekend, next weekend, Christmas, New Years, 4th of July, Labor Day, and pretty much every day in between. Pretty good for HDW not having a consumer-oriented business model.

Dang, I told Dave I was going to stay out of this. One more inaccuracy.

Ron Tiller
HDW

Not rolling in dough but a little flaky.

[andicap]

In the end, it comes down to the same thing as the American economy -- Confidence.
We use the dollar bill not because of the intrinsic value it has -- it's just a greenback after all -- but because of confidence we all have in its value, that it will be backed up by the full faith and value of the American government.

Now I'm not saying HDW data is backed up by the same level of confidence as the dollar (tho it doesn't shrink in value either), but I consider it a valid comparison. If I -- and hundreds if not thousands -- of other handicappers did not have confidence in the HDW numbers and their accuracy, their value would plummet.
In the end, anything in this game comes down to confidence. We all knew that result charts are often wrong, but are we confident ENOUGH in
their accuracy to believe we can still make money at it. Yes, for many of us. No for some of us.

Because enough other people have had success using HDW and have not complained about problems with their data, I have CONFIDENCE in paying for it.

I don't see how providing specs and other geek-talk would make a difference.

Having been involved with Bill Burns and Michael Perry when they dealt with HDW and heard how hard HDW works to get it right and used their data as well, I have an enormous amount of confidence in them, more so than say, the Daily Racing Form. I mean does the DRF put out any specs for accuracy?

I think some people here are just always looking for an argument. Being skeptcal is a good thing, but there's a fine line between skeptical and cynical.

[Secretariat]

As to file specs, yes Bris, TSN ,the DRF, Pizzola's group and Trackmaster provide easy access to file specs. Trying to get them from HDW is a nightmare.

[VetScratch]

I have no major disagreements with what Ron Tiller posted about technical accuracy considerations except to point out that accuracy (like content) is a value-added attribute, and all of the value-added data vendors do a poor job of conveying data value to customers in exchange for money. Of course, I will subsequently defend this assertion.

However, about the "morphing" referenced by Ron, here is how Ceejay started this thread: "I'm just curious because all software is only as good as the input data (you know, garbage in-garbage out), do users think that HDW data is better or worse than other data sources such as TSN/BRIS, DRF, ...? If HDW is better, why? Is there better data QC @ HDW than the others?"

The question was not literally constrained to accuracy, and the reference to 'QC @ HDW' is parenthetic to the rhetoric condition that HDW might be better than others. Ceejay later said his intended focus was "accuracy," but the thread was out of the barn by then (and I assume some deference was being accorded to Dave's sensitive nature since Dave had already "contended" that only "accuracy" was within the scope of this thread). In fact, "accuracy" was introduced by Dave rather than Ceejay.

I like Dave in most respects, but I wonder why he accuses others of "word games?"

Dave certainly offers us his share of word riddles. With respect to HDW data accuracy and his role in this thread, he says:

"HDW is squeaky clean data. Not just good but ALMOST perfect."

"If you check most of my posts I think you will find that historically I am a horse player and PA member first and a vendor second."

"If HDW had data problems (and I was still doing business with them) I would simply have been quiet about it."

I find ample candor in these statements, but "situational objectivity" is not the same as "absolute objectivity," so I can't understand why Dave expects anyone to confuse him with the village priest... he is a vendor, pure and simple.

I chose to enter this thread to discuss the fundamentals of the value-added distribution channels as a way of contributing to this board. My oberservation has been that questions and debate about the merit of data often fail to transcend partisanship and anecdotal experiences. More often then not, when threads like this mercifully expire, users are not much better equipped to be discriminating consumers than they were prior to a polyglot of conflicting opinions.

I assert that none of the data sellers merit praise for the business model offered to consumers. I further propose to show that the value-added pipeline leaks like a sieve insofar as the measure of value conveyed to consumers in exchange for their dollars.

The obstacles to evaluating what data is good, better, or worst are not constructs erected by users. Quite simply, the data and software vendors inhibit users from becoming discriminating consumers.

As I posted earlier, the beginning of my week is largely allocated to other matters, but I fully intend to pursue this matter... by the end of the week... and demonstrate how the fair measure of data value, with respect to both content and accuracy, is not making its way to consumers.

Tom and Ron Tiller have offered excellent and informative posts. I especially appreciated their posts because they have provided examples that will help me illustrate my case.

[RonTiller]

To Vetscratch,

Yikes, this is starting to sound like a graduate seminar:
"fundamentals of the value-added distribution channels" sounds like a good title for a master's thesis. I'm not so sure about "situational objectivity" and "absolute objectivity" but I've always been philosophically cantankerous anyway (you know, you never actually see OBJECTS, you only see the SURFACE of objects, etc.), so I won't press it.

I myself entered the thread in an attempt to advance the topic and your observations about the ultimate and merciful fate of threads like this are probably right.

Your statements, soon to be expounded on, are no doubt deliberately provocative and it sounds like they would be equally appropriately delivered at an industry symposium, like that going on in Tuscon right now. I'll be happy to contribute any thoughts or knowledge I have on these matters once you post them. I am not, though, going to try to justify our business or our business relationships or our relationships to our customers with anybody on this board. If this is expected, I will gracefully leave the proceedings with no rancor whatsoever. The thread must go on.

Perhaps once you amass your case, you could start a new thread. I like "fundamentals of the value-added distribution channels".

To ceejay,

I don't know that you really got the answer you were looking for, with all the various point/counterpoint, personal attacks, deconstruction, sematic analysis and apologetics going on. This is a lively group for sure.

And to All,

A goodnight.

Ron Tiller
HDW

Not even situationally objective!

[ceejay]

Thank you Ron, and others for the input. I was able to get the basics behind my question answered. Liberal use of the Ignore feature keeps my mind from wandering too much.

[Tom]

Originally posted by VetScratch
Tom,

Have you considered what would make you happier than you are?


Yes. But HDW does not offer it!:rolleyes:

[karlskorner]

This was mentioned a couple of posts back, but no one picked up on it. About 80% of the information provided by data suppliers comes from the chart caller at a particular track. 1, Are we to assume that the chart callers at a Class 3 track are equal in their craft to a chart caller at a Class 1 track, not unless Equibase cloned a lot of people. 2. There are usually 2 people making a chart, the caller and chart maker, sometimes/almost always they alternate, so it is possible on a 10 race card that 5 races were called by Fast Eddie and the other 5 by Slow Sol, you now have two opinions as to the position and lengths back at one track and the opinions of all the chart makers at all the tracks running on a particular day. 3. The timing devices at all the tracks are not identical and some malfunction, so a 1/4 time of 22.3 at one track could be 23.1 at another. As "clean" as the information provided by data suppliers is, they like the Beyer numbers that are adjusted by his 5 employees, are controlled by a human being

[takeout]

Does anyone remember the old chart books? I never have seen one as they were a little ahead of my time but I was curious if they had charts for ALL the tracks or just some of them?

[plainolebill]

They had charts for every track in the country. Came with a seperate index book.

[VetScratch]

Karl, Plainolbill, & Takeout,

How about the old DRF charts on microfiche? Over the years, my dad amassed shoeboxes full of microfiche charts... every race in every year. With a little practice, you could really navigate quite rapidly to find any chart you wanted to study.

Does DRF still offer the microfiche charts?

[VetScratch]

Ron Tiller,

Provocation is not my agenda. I certainly have no intention of provoking horseplayers except insofar as motivating everyone to become a discriminating consumer.

My issue is consumer advocacy where handicappers are the consumers. We debate and discuss consumer issues all the time with respect to racetrack operators, wagering services, pari-mutuel technology, and trainer/jockey ethics. Why should the data sellers be exempt from examination and criticism?

I simply maintain that none of the horseracing data re-sellers offer us a satisfactory deal.

Instead of handicapping races, if the task was research for Toyota, we could expect to deal with several data sellers. We might buy refined census demographics from Firm-A, auto registration data from Firm-B, auto dealership data from Firm-C, and media statistics from Firm-D. The original sources of data from Firm-D might be regional market research firms that compile and sell data about advertising in printed publications, on radio, and on TV. By evaluating all purchased data and Toyota's own dealer database, we could "handicap" the automobile market and "post our picks" for Toyota: add more dealers in Illinois, buy more radio spots in Texas, emphasize luxury models in Seattle and Boston, etc. We might also leverage our data purchases by selling our "handicapping" services to Nissan, Ford, etc.

All that I intend to discuss are considerations that make data commerce a square deal for both sides of a transaction. Most of the benefits that handicappers fail to receive could be provided by vendors with little effort and at insignificant costs.

By Sunday, I will carve out the time to finish posting on this topic.

[takeout]

Originally posted by VetScratch
Does DRF still offer the microfiche charts?
Pretty sure they don't. That was probably two or three owners ago.

I used to see those ads and want to get them but I never actually did. I've never been more than a local player but they would've been nice to have. As far as I know there is no viable alternative for them these days.

[Jaguar]

VetScratch,

good strong post. You have a first class mind as well as good values.

All The Best,

Jaguar

[Gold Bay]

I'll second that. Ive lurked here daily for a few months and recently started posting.VS i have always enjoyed your posts. They are very insightful ,you obviously know a ton about this game . Thanks for your time and effort.

[VetScratch]

Electronic Data (edata) Overview
Annual edata costs for many horseplayers will exceed what they paid for their computer. For $1200 to $5000+ per year, players should expect standards of service that approach business-to-business edata relationships. Instead, providers of racing edata offer a very casual level of service to players. Because the cost differential between good and shoddy edata service is insignificant, unsophisticated consumer expectation is probably the reason why service is shoddy.

Whereas edata offer more potential value than printed data, horseplayers have not insisted that providers deliver the full measure of potential value. Acting as shrewd consumers, players certainly have the economic clout to dictate better standards of service, which providers would be able to adopt without incurring significant costs.

Like most high-technology products, good edata should convey more value than the average consumer may use at any point in time. Then, as consumer sophistication increases, good technology products are always enhanced to stay ahead of the curve. In the usual case, upwards of 90% of users derive passive benefits from the activities of advanced users, who act as consumer watchdogs and keep the user community informed. We do this here on PA, but discussions are clouded in comparison to other industry newsgroups because horseracing edata vendors have been reluctant to provide truly comprehensive product information.

Let us consider what could be done to improve edata service for horseplayers that build databases and/or use handicapping software. These players usually want two edata products:
(1) Racecards: traditional PP data and (optional) value-added data.
(2) Results: summary race results or full result charts.

Three aspects of edata service seem to beg for universal improvement:
(1) Unfettered Access.
(2) Product Information.
(3) Rights/Restrictions.

Unfettered Access
If we buy an edata product, we should expect to download it to our computer and have unfettered access to the data content. Instead, many edata products are devalued by artificial constraints and marketing gamesmanship.

When edata is downloaded by users, the only barrier between edata products and users should be compression and decompression packaging such as the industry-standard ZIP specification, which provides rigorously tested safeguards against data loss and/or corruption.

Good edata products deliver maximum value when they are downloaded to your computer as data files that may be accessed in a variety of ways:
(1) Direct access by viewing programs such as past performance viewers.
(2) Direct access by any number of commercial handicapping packages that elect to support a specific edata product.
(3) Direct access by programs that you (or anyone else) may develop to handicap racecards and build databases.
(4) Direct access by data exchange software that extracts validated data in formats that are compatible with mainstream commercial products such as MS Access or MS Excel.

Unfettered direct access to edata products means that no artificial barrier or filter stands between users and edata products. The entire user community benefits when edata products offer unfettered access:
(1) Data can be used in more ways by any user, resulting in the development of more means to use any edata product.
(2) Data will be validated by more means, resulting in broader quality control feedback and better product integrity.
(3) A better informed user community will emerge to provide objective peer-to-peer support and meaningful product evaluations.

Since almost all race-results data and the bulk of racecard data is common to all handicapping edata products, any impediment to unfettered access must be viewed as a defect or a marketing ploy that devalues an edata product.

Racecard edata products consist of "traditional" data "common" to all such products plus varying quantities of additional value-added data. In recent years, commodity pricing has caught up with racecards that contain very little value-added data. For commodity-priced racecards, Trackmaster/Equibase offers conversion software that creates BRIS/DRF racecards, and BRIS could certainly elect to likewise offer software to convert BRIS/DRF data to Trackmaster/Equibase racecards. However, while such conversion programs expand the number of ways that users can make use of racecards, conversions are potentially undesirable, just as filters are undesirable.

The most desirable delivery package for edata racecards would be a ZIP file that contains components which clearly separate traditional/common data from value-added data. For example, a racecard ZIP file might contain three components:
(1) traditional/common PP data from Equibase or DRF.
(2) value-added data created by Equibase or DRF.
(3) value-added data created by a reseller such as BRIS, HDW, ITS, TSN, etc.

Such a product delivery package, properly documented, would make it easier for consumers to differentiate between products, and would reduce confusion and finger-pointing when we gather on message boards like PA to discuss and critique edata products. I have seen posts that pin the tail on the wrong donkey because users don't always know whether a data element was created by Equibase/DRF, by BRIS/HDW/ITS/TSN, or by a commercial handicapping program. Since ambiguity escalates customer service costs and clouds the reputations of edata providers, we are naturally tempted to question the motives behind confusing and shoddy edata product offerings.

By filters, I mean mandatory obstacles on users' computers that stand between edata products and users. Filters obstruct users' ability to validate edata products and may also constrain users' ability to effectively use edata products. Except for straightforward decompression functions (e.g. un-zipping), a filter is any mandatory program that you have to run to gain the unfettered use of purchased edata. Thus, filters include any conversion program, racecard viewer, or handicapping program that you must run to "extract" or "export" data from edata products that are delivered under circumstances that restrict the unfettered use of purchased data. I have yet to see a mandatory filtering scheme that does not devalue horseracing edata products in one way or another.

Data from racetracks has always been subjected to scrutiny and correction before going to the printing presses. Given the way racing information is originally recorded, the accuracy of printed products was always more remarkable than the error rate. Today, we should expect even better accuracy because powerful and tireless computers are available to perform sophisticated error checks at each stop in the edata distribution chain. Unfortunately, professional standards to facilitate validation of horseracing edata are seldom extended to product users.

In a distribution chain, each computer stop is always a work-in-progress. Computer hardware and software environments are modified and upgraded on a continuous basis, including your own personal computer. Naturally, bugs creep into complex systems on a continuous basis.

Consider how edata validation is supposed to work in a value-added distribution chain, using horseracing results as an example. Race results include payoffs and final odds that eventually feed a database on your computer, which is maintained either by you or by commercial handicapping software. Payoffs and odds should be validated after they are reported from tracks and before they are redistributed in any of the many race result products offered by Equibase and DRF. Since pool breakouts, takeouts, and breakage rules are not validated and distributed with results, payoffs and odds should be distributed with documented validation criteria that help resellers re-validate what they buy. Resellers need to re-validate payoffs and odds at least to the extent that errors caused by software bugs are not likely to be inherited from Equibase and DRF. Likewise, payoffs and odds that reach your computer should have documented validation criteria to help you or commercial handicapping programmers re-validate to the extent that errors caused by software bugs are not likely to be inherited from the resellers.

CONTINUED...

[VetScratch]

It is tricky but not exceedingly difficult to validate payoffs and odds so that gross errors are easily detected. Final odds lines can be checked against known takeout/breakage limits, and then WPS payoffs can be checked against final odds with enough precision to detect any gross errors. The same is true for other data within edata products for racecards and race results. However, the horseracing edata industry stops passing on documented validation criteria before edata products reach users. This would not be tolerated in a business-to-business distribution chain, even if annual subscriber costs were as low as $1200 to $5000+ per year.

Because industry standards are lax, the quality of data validation in commercial products is lax. In all of the posts concerning errors, it seems that glitches only get discovered when they are observed by users. This is how printed errors have always been discovered, but it is hardly a professional approach to validating edata and detecting software bugs in the distribution chain.

Here is one example. The most popular race results file in the industry is used by dozens of commercial handicapping products. Between 1998 and 2002, payoffs were grossly in error whenever there was a dead heat for win, place, or show. During these five years, I cannot recall any users of dozens of commercial programs reporting that this error got caught by their commercial product. How can dozens of commercial software authors miss gross errors? Simply because they follow the leadership of edata providers. Lazy error checking standards stem from the fact that edata products are not professionally documented by edata resellers.

The additional problem with mandatory edata filters is that they stand between users and edata products. All true filters, including popular viewers and handicapping products are susceptible to bugs or lax standards for data validation. For example, I discovered several years ago that when you download Formulator past performances, a secret password prevents you from directly accessing Formulator data (in the zip-file downloads thinly disguised as something else). Thus, while Formlator is a very fine product, it is still an example of unnecessary and undesirable filtering. In many other ways, many other commercial products act as filters... susceptible to bugs and usually offering to extract only a subset of the data purchased by users. This kind of service seems to be encouraged by edata providers, but it does not serve the best interests of the user community.

I will post again to finish discussing the edata industry with respect to:
(1) Product information shortcomings.
(2) Ambiguous or confusing rights/privileges conveyed to users.

[Tom]

You are fitting the model of a disatisfied customer, or a customer who traditional sellers are not marketing towards.
This would be the time for the bitching to stop and the entrepenurial spirit to take over. In other words, you don't like the
way the game is being played, change the game. Go into business and offer what you have described.
Hey, I would prefer it is cocktails and desert were included with all diners. Should'nt I have full access to the kitchen when I go out to eat? Ins't selling entrees bad business?
Why should the place not offer my unfettered access to the ovens? And shouldn't I be able to take home as much as I want as left overs from an all-you-can-eat buffet?

[VetScratch]

Not quite, Tom... but you should be allowed to eat everything on the plate that you buy! :)

[Tom]

Originally posted by VetScratch
Not quite, Tom... but you should be allowed to eat everything on the plate that you buy! :)

Touche' :p

[Handle]

VS --

I don't think one needs a 30 page essay to say that a data standard for horse racing information, that EVERYONE adhered to, would be beneficial to consumers. But in your championing of the consumer here you've conveniently neglected a thing or two.

Most importantly, you haven't offered any valid reason why data providers who do publish their data in proprietary formats, or software producers who partner with these providers, should do otherwise. I've seen these relationships exploited in ways that could be called a "marketing ploy" (Free Software! (but 100$+ for data!)). In general, however, I don't think the deragotory "ploy" is called for -- I'd call this a business strategy aimed at protecting ones marketshare. You, or any consumer, may not like this, but no one is going to give up revenue and control of their product in the name of consumer convenience.

"Unfettered access" to the data from the few data providers we have today is NOT a way to gain higher quality data products. That is a pipe dream which would result in having even fewer data providers because the smaller ones -- the one's not as successful at true "marketing ploys" -- would quickly go out of business. In the absence of other data providers there would be even less incentive to produce this quality data you are looking for.

The only thing that could drive this forward would be an UBER software program that controlled most of the marketshare of software handicappers. And then this uber program would have to declare that it would use data in a certain, open, format. Then the smart data providers would see which side the bread was buttered on and develop data files for the program.

The above is not what's happening - instead its the reverse. The data providers dictate the terms. The reason, I believe, is partly due to the lack of free enterprise in the data providing game. If you had more of a free enterprise -- if EQUIBASE made its _raw_ data more easily available to anyone wishing to become a reseller -- the quality of data would most certainly improve as competition has a way of improving everyone's product. I don't know everything about this situation, but I have looked into it a bit. This is what I believe to be true:

Despite the fact that we live in a day and age where Equibase could provide resellers "raw" data at little to no cost:

A) Last I looked they had no plan where one could just sign up to become a reseller of their data.

B) Any deal made to resell the data would have to be made "on the inside", and/or, wouldn't be cheap for the person wishing to start reselling EquiBase data. They'd want up front cash and/or monthly commitments. But this is all mute if they won't work with you to begin with.

Overall, my impression is that the economics of the horse racing industry -- on the data and software supply side at least -- is very fragile. Where it thrives it does so on trusts between entities and on reputations alone. This does not always lead to the best thing for consumers, or even the most profitable ways of doing things, but it probably allows the people in this side of the game to keep on playing.

[Tom]

Don't you just hate it the way they sell you a car and then you have to put gas in it every week just to drive the damn thing????Sheeez! What a racket.
:rolleyes:

[Jaguar]

VS, as one of your enthusiastic fans I want to comment on a couple of points you raised recently.

First, Dave Schwartz is a really smart guy, an honest guy, and a very talented programmer. I began using his masterpiece- Thorobrain- in 1994 and I can tell you that it was head and shoulders the best A.I. program(and indeed, the best handicapping program- period- on the market at that time).

In other words, Dave actually does write code.

Secondly, I'm curious that you commented that motorcycle cops and bullriders are essential to fantasy- since I- as a healthy adult male- have never found these images appealing in the slightest.

Rather, I find that the concept of a week-end at Caneel Bay with Carmen Electra is far more exciting.

The volcanic sand there is so fine, a pleasure to walk in- feels like silk- and that water, pure aquamarine and crystal clear.

I wouldn't need a tv in the bungalow, I can assure you.

All The Best,

Jaguar

[VetScratch]

Handle,

I never proposed a standard data exchange format... just an optimized way to organize it so consumers could better grasp the difference between edata products (i.e., organized to show where the various data items originate). This would be nice, but good product documentation will accomplish the same purpose.

Is the following really such an enigma?

The edata providers should compete on the basis of:
(1) Product Price.
(3) Product Documentation.
(3) Product Content.
(5) Product Utility.

A contractual monopoly at the front of the distribution chain helps EquiBase and DRF, but there is plenty of room in the value-added sector for product differentiation and tiered pricing.

You seem to think consumers should disregard documentation, content, and utility when shopping for data.

So far, I have discussed primarily utilility (i.e., ease of use and verification). Why do you think restrictions on use help sales? For today's racecards and results, what is served by discouraging the use and development of many ways to use the data? The answer is not one that serves the interests of consumers.

You fail to impress me as a consumer advocate.

[lousycapperII]

After reading this thread I have come to the conclusion that perfection is not in the handicappers vocabulary. If all data comes from Equibase then all data is correct or it is flawed. We all use this data in our own way; so we are all correct or we are all flawed. Just gimme single comma delimited files or the like and I'll do just fine regardless of who's supplying the data.

-LCII

[Tom]

If anything you suggest would improve sales, don't you think it would have been done already?
If you need all the things you list, you constitute a small market. What is in it for any of the sellers to do anyhting you suggest?
Do you really think HDW is going to sell more data?
Do you rreally think TSN will sell more data?
BRIS, TSN, and DRF offer tons of data in their files and you can get the documentaion on the websites. No one else seems to be having a problem with what they get.
HDW sells dazta to support specific programs, that in turn convert that data into proprietary figures which we can download.
Just what particulat bit of data is anyone witholding from you?
We are provided with time utility and place utility everyday. WE have the programs to use the data, and the means to export from it. Just what the hecl are you missing?

[VetScratch]

If there is something in it for the consumers, why is a critique of the sellers upsetting to any of you? :) :)

Better consumers make better sellers.

[VetScratch]

Tom,If anything you suggest would improve sales, don't you think it would have been done already?Framed from a consumer's point of view, shouldn't we be asking why sales might not be improved?

Is there a reason we can't open the hood, look in the trunk, start the engine, take a test drive, or have our mechanic look at it? Yes, there probably is.

[VetScratch]

Jaguar,Dave Schwartz is a really smart guy, an honest guy, and a very talented programmer. I began using his masterpiece- Thorobrain- in 1994 and I can tell you that it was head and shoulders the best A.I. program(and indeed, the best handicapping program- period- on the market at that time). In other words, Dave actually does write code.That's what I hear, but I am naturally apprehensive about the HDW/HSH business model when a 30-day-trial has become such a successful formula for most small software vendors... why should horseracing be different? Even big vendors are adopting this model. The expensive TurboTax Professional Edition offers a free trial, as do many other companies that believe a superior product sells itself. Only horseracing seems to cling to outmoded marketing paradigms.Secondly, I'm curious that you commented that motorcycle cops and bullriders are essential to fantasy- since I- as a healthy adult male- have never found these images appealing in the slightest.I was playfully replying to an ex-cop and motorcycle enthusiast... but bullriders are ideal because you don't have to get rid of them; they just naturally ride off to the next go-round. No need for maudlin goodbyes. I just say something like, "Honey, you take care... don't dwell in the well or get whipped down so bad that we can't use you if I see you in Cheyenne." :)

[Handle]

Originally posted by VetScratch
I never proposed a standard data exchange format... just an optimized way to organize it so consumers could better grasp the difference between edata products.


Then I must have wildly misunderstood when you suggested that:

"When edata is downloaded by users, the only barrier between edata products and users should be compression and decompression packaging...."

And that

"Good edata products deliver maximum value when they are downloaded to your computer as data files that may be accessed in a variety of ways:
[...]
(3) Direct access by programs that you (or anyone else) may develop to handicap racecards and build databases."

OK, that spells documented format to me. Then you add that:

"Unfettered direct access to edata products means that no artificial barrier or filter stands between users and edata products."

That spells standard format to me, since who'd want a "filter" to get at that data?

I didn't get any notion about a desire to compare products from your post. I read a desire to be able to use data in the widest manner via "unfettered access". This, I pointed out, is not in the best interest of the people selling the data for a variety of reasons that are clear and obvious, while not always consumer friendly (the goal of a business is to produce a profit, not produce friends. It is only coincidence that the two sometimes go hand in hand). I'll retort your rhetoric - is this such an enigma to understand?

Oh, but yes, what you suggest is an enigma to understand when you cloyingly clutter your simple meaning with collections of meaningless verbiage. For this you dissapoint me as self-appointed spouter of the truth.

As far as assessing the quality of the data on a comparitive basis - I think authors of programs that use proprietary "closed" data would tell you to evaluate the quality of their products in conjunction with the data -- its a package deal, after all. Yes, the "our data is better" card is often played to somewhat of a sham, I agree (and I call it when I see it), but its not something to warrant a diatribe the likes that you are in the midst of producing. In short, I found your post to be a bit on the reckless side given the "artificial constraints" and "marketing gameship" cards that you are playing here.

The bottom line is that there are more powerful aspects of fairness in the market and what you perceive to be "data quality" issues that enter the picture far before any question of consumer advocacy. To pick on a data provider and/or software producer because they won't make their data generally pillageable by other products, or products pilleageable by other data providers, is simply barking up the wrong tree.

If you're asking for data that is documented - it exists. Those that feel the need for documented data will obviously demerit products that are not documented - why do you need to attack them as if this fact eluded the consumer? It does not.

If you want data that can be most widely used "as is", the choice is clear as well - find the least costly data that the most software is written for.

If you're asking about "content", then that is evident by the data that is documented and/or by the programs that add value/content to the data - I'm sure their authors will tell you about what their products provide.

Price is self-evident as well.

Given all this I indeed see no enigma. In its stead l find little need for this investigation of yours - there is only your complaint - mishappen, uninformed, and generally aimed at the wrong target, I'm sorry to say.

[VetScratch]

Handle,

What have I suggested that would invite criticism from handicappers?

The way you propose that consumers should shop is analagous to asking a furnace manufacturer to recommend a choice of fuel (i.e., gas, oil, or electricity). Are you seriously proposing that consumers should ask the software vendors for an objective evaluation of the data providers?

I find it unlikely that a gas furnace manufacturer will ever recommend oil or electricity.

[Shacopate]

I think you should change your handle to "HELLCAT".

No disrespect intended (it might actually be a compliment).

It seems that there is smoke and fire wherever you post.

[GameTheory]

I vote she change it to that delightful phrase from the Seabiscuit book:

"Shit Godzilla"


I can't think of anything more appropriate...

[VetScratch]

Shacopate,

I think some folks are getting prematurely alarmed.

When the whole class gets a "D-" only the consumers should be disappointed. :)

[VetScratch]

Hi Game,

I am merely proposing that edata vendors voluntarily make it easier for consumers to select and buy products, obtain maximum benefit after the purchase, and clarify the terms of use.

This would discourage hackers from inviting consumers to avoid purchasing data by dubious means. Does your utility export the copyright notice from the HTML result charts?Proprietary to and (c) 2003 Equibase Company LLC. All Rights Reserved.
Republication or dissemination of the contents of this screen are expressly prohibited without the written consent of Equibase Company LLC.
Use (including viewing) of the material contained herein constitutes acceptance of these terms.

[Shacopate]

Once again,

Smoke and fire.

The only question I have is why?

[VetScratch]

Shacopate,

For the same reason we:

(1) Question takeout and wonder where it goes.
(2) Question drugs and other forms of cheating.
(3) Question money coming into pools during and even after horses.
(4) Question the practices of NYRA, Magna, and Churchill.
(5) Question whether slots will save or doom racing.
(6) Question how horses meet their maker.
(7) etcetera ad infinitum.

How is the edata business any different?

[Shacopate]

I was referring to your dig on Game's parser. A slight, but still back handed slap.

Do you work for the government?

[VetScratch]

Shacopate,

Game asked for "Shit Godzilla," and he found her. :)

[Shacopate]

And I'm sure he'll get more than that.

Just don't understand your agenda.

[VetScratch]

Shacopate,

Game did nothing illegal by writing the result chart utility.

Only users should ponder whether they are playing by the rules.

[GameTheory]

As usual, VS missed the point. My suggestion for a handle change has to do with pretty much all her posts, not just this thread. But I'll shut up now, and you can go back to talking about "edata"....

[VetScratch]

Handle,
Originally posted by Handle,
To pick on a data provider and/or software producer because they won't make their data generally pillageable by other products, or products pilleageable by other data providers, is simply barking up the wrong tree.How can users possibly pillage what an edata vendor has sold and delivered to their computers? Are you suggesting that lack of documentation, or poor documentation, is the industry's answer to piracy? It seems to me that selling edata without documentation is almost a contradiction of purpose. What do we buy edata for if not to understand it and make use of it for handicapping? Should users be deprived of good product information because edata providers fear piracy among themselves?

[Tom]

The horse is dead.
Stop whipping it. :D