Has Twin Spires Crashed ?

[rcknhrse]

cant get on using fire fox or int exp

[kev]

I can't get in either. I already got all my bets in early on.

[DeanT]

Hmm, this sucks. I dont have a TV where I am at.

If anyone knows where else I can watch it on the web let me know. Sure glad I switched over the feed to watch the 9th at Belmont.:bang:

[rcknhrse]

I wonder if its sabotage or their server is just overwhelmed

[pdxmike]

i found the nbc feed here...http://channelsurfing.net/

[Donnie]

it's toast

[Oddzilla]

I guess they didn't plan on doing so much business on the Kentucky Derby.

C'mon twinspires, you can do it--more power!

[Tom Barrister]

As of a minute ago, it's not working. Fortunately, I got my bet (boxing five longshots) in early.

[Oddzilla]

It's gone again. Bodog is spinning it's wheels too. I made bets earlier, but want to get some savers in.

[Tom]

Biggest day of the year and theses pathetic morons can't handle it.
As soon as they come back up, I'm closing out and they can kiss my ass.

[rcknhrse]

allhorseracing.com is having trouble also

[shanta]

Biggest day of the year and theses pathetic morons can't handle it.
As soon as they come back up, I'm closing out and they can kiss my ass.


:ThmbUp: :ThmbUp: :ThmbUp:

Ass Backwards ( track execs,horsemen,Adw's etc) strikes again man :ThmbDown:

[HEY DUDE]

WTF!!! I'm done with these clowns. Last night I was on hold for an hour trying to make a deposit that I cant use now. DONE

[Oddzilla]

I am so fricken pissed off! Unbelievable.

[Tom]

flood them with emails complaing of their totally incompetant performance.
I have sent 25 so far. :ThmbDown::ThmbDown::ThmbDown:

[GameTheory]

This happens every year. The servers are overloaded. Gotta bet early on Derby Day...

[cj]

This happens every year. The servers are overloaded. Gotta bet early on Derby Day...

Funny, we all know it happens, and they never fix it.

[Donnie]

It's back up. We can all go bet again..... :bang:

[Valupix]

Probably saved a lot more people some money, than it did those that would have cashed had they gotten in.

Who many times have you been shutout and mad as heck at the teller, only to minutes later be overjoyed that you were able to get in what would have been a losing bet.

[rcknhrse]

should have listened to twinspires to get your bets in early .actually their advice was to fund early.
did not anticipate this crash happening.
i know last year i got in near postime

[Tom]

This happens every year. The servers are overloaded. Gotta bet early on Derby Day...

Actually, no, we don't.
We don't have to bet at all.
If this bunch of morons can't figure how to take bets - their core business - it might be time to find a new game, one run by people with IQ's over 7.

That's like saying it's busy during the dinner hour, phone in your order in the morning.

Like I told Tom in another thread, this is hardly the only game in town. They just think they are.:ThmbDown::ThmbDown::ThmbDown:

[ceejay]

This happens every year. The servers are overloaded. Gotta bet early on Derby Day...
Hasn't happened for years when it was americatab actually.

[DJofSD]

Funny, we all know it happens, and they never fix it.

Will we ever know exactly what happened? No. There will be something published about it but it will not have any specifics, i.e. something that an experienced IT person would be able to read and understand that likely was the root cause of the problem. A press release of "unexpected heavy demand" is just not going to cut it.

Very likely a critical server did get over loaded. Here's a plausible scenario: b/c TS was not taking any bets on other races on the card, they did not have adequate resources when the flood came during the betting period leading up to the race.

Management infrequently will plan for and fund expenditures to be able to handle very dynamic peak loads. If they can get away with just enough to handle the normal load, that's what is approved. But to spend money on a maybe, to have the utilization numbers show a very low level of usage all the time and to have that additional equipment just idling, they take their chances and call it risk management.

[Fingal]

This happens every year. The servers are overloaded. Gotta bet early on Derby Day...

If it crashed I can only hope that they lost my losing wagers & they're floating somewhere in cyberspace:lol:

[GameTheory]

It didn't crash. It just became really slow because it was overloaded. All the betting sites seem to suffer this on every big racing day. It may not even be fixable -- they can't control all the networks and routers after all, only their own servers. More likely is that they could fix it by distributing lots of servers in different locations, but the expense would be so great to upgrade the capacity that it isn't worth it for 2 or 3 days a year when it is a problem. Or maybe they have upgraded and it just wasn't enough. There is also really no way to tell how much capacity you need unless you already have too much. And they won't be able to find that out until next Derby Day, although Breeder's Cup or the Belmont if you've got a possible TC winner are also very big.

Once you are at your limit I don't think there is a way to tell how much traffic you are rejecting. Let's say they are right this minute trying to figure that out for next year -- I'm not sure they can tell how much more capacity they needed -- only that they needed more. (BTW, this is an AmericaTab issue, not TwinSpires per se. TwinSpires is just one affiliate of AmericaTab. So CD is not to blame here unless the host track itself is causing the bottleneck -- possible I suppose.) From their point of view, everything probably looked fine -- they took all the bets they could take. There didn't seem to be any actual crash or downtime. (I managed to get in some late bets, but had to try several times.)

[ceejay]

Let's say they are right this minute trying to figure that out for next yearI would not be so sure of that.

[GameTheory]

I would not be so sure of that.I did say "let's say" -- point being is that there is probably no way to tell how much more capacity they need...

[NoCal Boy]

Churchill should be embarrassed. They had two races today and the system all but shuts down. What the hell would have happended if twinspires had all the races today? Go out and get a platform that works at your peak times.

[Tom]

The capacity this mornic industry needs is to allow everyoen to take waeers onthe big events and not hog them all to themselves. There is no acceptable reason why Youbet couldnot take Derby wagers and provide a service to the people who pay the bills - us. Not he friggin tracks, not the friggin horsemen - us.

I have already writen my congressman and expressed a dire need for governement oversight and encourage everyone to do so. Let the horsemen deal with them.

[richrosa]

GameTheory,

I don't mean to disagree with you, however having run or consulted for major eCommerce or content websites, I'd like to set the record straight, so that my fellow consumers of ADW products understand what they are purchasing when they place their money and trust with a product.



It didn't crash. It just became really slow because it was overloaded. All the betting sites seem to suffer this on every big racing day. It may not even be fixable -- they can't control all the networks and routers after all, only their own servers.

There are many components that effect the operation of a high transactional website. Certainly one of them is general internet traffic (Saturday after noon = very low internet traffic or 10% of weekly peak), and the local internet traffic aka the size of the pipe to their hosting center. With bandwidth being cheap these days, there is not excuse in not having multiple BIG pipes (I'll stay in laymans terms to keep it simple). I'd like to believe that the problem was not bandwidth today, and if hosted in a qualified data center, you can be assured that it wasn't the problem.

Regular eCommerce businesses before holidays like Christmas, or Valentines Day in the flower business, or the World Series in baseball, or any kind of surge event, generally have a capacity planning exercise where the projected traffic is tested against the platform in an effort to understand how the environment will perform when the cash register is really open. These exercises result in enhancements and upgrades, and eventually a validation that ensures that the projected traffic will be serviced properly.

To say that it may not be fixable is completely not true. It is very likely that they have a sub-standard architecture that either did not pass a capacity planning test, or there might be amateurs running the operation who didn't bother to do a capacity planning test. Fixing a problem like this may mean an investment in the architecture, which I bet at some point was recognized as needed (like last year when it melted), and bypassed because, we the customer don't mean dirt to these firms.



More likely is that they could fix it by distributing lots of servers in different locations, but the expense would be so great to upgrade the capacity that it isn't worth it for 2 or 3 days a year when it is a problem. Or maybe they have upgraded and it just wasn't enough. There is also really no way to tell how much capacity you need unless you already have too much. And they won't be able to find that out until next Derby Day, although Breeder's Cup or the Belmont if you've got a possible TC winner are also very big.


I agree with some of this. they could redistribute their load to many locations. This would solve a bandwidth problem, and possibly overcome some architecture inefficiencies. I do disagree that they won't be able to tell until the next holiday. There are many companies and products out there that ssist in capacity planning. Its more likely that its amateur hour at TwinSprires.


Once you are at your limit I don't think there is a way to tell how much traffic you are rejecting. Let's say they are right this minute trying to figure that out for next year -- I'm not sure they can tell how much more capacity they needed -- only that they needed more. (BTW, this is an AmericaTab issue, not TwinSpires per se. TwinSpires is just one affiliate of AmericaTab. So CD is not to blame here unless the host track itself is causing the bottleneck -- possible I suppose.) From their point of view, everything probably looked fine -- they took all the bets they could take. There didn't seem to be any actual crash or downtime. (I managed to get in some late bets, but had to try several times.)

Your border routers can tell you how many connections are being requested, and the web platform should have traffic monitoring to do the same, so they should have a clue as to how much customer traffic isn't being serviced.

CD is to blame. They are the parent company for TwinSprires which purchased the AmericaTab platform before Derby Day last year when they recognized that their own internal development effort to build an ADW platform had failed. Occasionally there are tote bottlenecks that cause problems, however there is not reason to believe that happened today. Most likely the AmericaTab platform (which is over 10 years old) was never written to scale to todays usage levels and needs an overhaul itself.

Just to keep tabs. I have read here that XpressBet had problems too. I tried to bet there 30 minutes before the race and had problems. We know that platform has many problems in the past. This morning I bet with NYRA and had problems. That site is built by Sona Mobile who I believe should know better about how to build a scalable platform, but has show repeatedly that they might not be qualified to build an ADW. I can't wait to see Nassau OTB's Sci Games effort, if it ever gets delivered. That'll be a dog too.

We as customers should get sharp and keep score as to how these bozos perform when we need them and force them to deliver solutions that at least are up and available when we need them as our time and money is precious.

[DJofSD]

rich, excellent post. I'd just add that as a publicly held corporation, the CEO has a fiduciary duty to prevent problems like this from occuring. I believe this represents a failure that the board and the shareholders should not tolerate. I'm sure they lost enough money b/c of not being able to take wagers due to the (illegal) action on the part of the horseman's group. Having those losses worsen due to an inability to take bets on the one race they were allowed to take is just salt in the wound.

[GameTheory]

There are many components that effect the operation of a high transactional website. Certainly one of them is general internet traffic (Saturday after noon = very low internet traffic or 10% of weekly peak), and the local internet traffic aka the size of the pipe to their hosting center. With bandwidth being cheap these days, there is not excuse in not having multiple BIG pipes (I'll stay in laymans terms to keep it simple). I'd like to believe that the problem was not bandwidth today, and if hosted in a qualified data center, you can be assured that it wasn't the problem.Probably so -- the non-betting stuff was working sluggishly but not nearly to the extent as anything transactional -- betting, checking balance, etc.

Regular eCommerce businesses before holidays like Christmas, or Valentines Day in the flower business, or the World Series in baseball, or any kind of surge event, generally have a capacity planning exercise where the projected traffic is tested against the platform in an effort to understand how the environment will perform when the cash register is really open. These exercises result in enhancements and upgrades, and eventually a validation that ensures that the projected traffic will be serviced properly.What if your projected amount of traffic severely underestimates actual traffic? Other "big days" in other industries suffer from these same problems, capacity planning or not.

To say that it may not be fixable is completely not true. It is very likely that they have a sub-standard architecture that either did not pass a capacity planning test, or there might be amateurs running the operation who didn't bother to do a capacity planning test. Fixing a problem like this may mean an investment in the architecture, which I bet at some point was recognized as needed (like last year when it melted), and bypassed because, we the customer don't mean dirt to these firms. I don't see how it could be "completely not true" when I qualified with the word "may", opening the door to be corrected by the likes of yourself. No doubt they need an investment in architecture, but if they really only need it for a couple of big events a year the question is can it be done in such a way to get the capacity on only those days when they really need it instead of having lots of underused hardware they are paying for the rest of the time? They have improved some -- they used to be sluggish EVERY weekend in the summer.

I agree with some of this. they could redistribute their load to many locations. This would solve a bandwidth problem, and possibly overcome some architecture inefficiencies. I do disagree that they won't be able to tell until the next holiday. There are many companies and products out there that ssist in capacity planning. Its more likely that its amateur hour at TwinSprires.That's likely, but capacity planning is no good if you can't really estimate the capacity properly other than to plan for the biggest day you can possibly imagine -- or more accurately the biggest MOMENT -- it did work most of the time, but as the Derby got nearer, the slower it got. (And was quite speedy after the Derby was over.)


Your border routers can tell you how many connections are being requested, and the web platform should have traffic monitoring to do the same, so they should have a clue as to how much customer traffic isn't being serviced.A clue, but still a lot of guessing. Lots of potential bettors probably get one or two time-outs (possibly just trying to login) and then give up. I can't see how they'd really know what is going to happen at the peak times unless they have the experience of actually having plenty of capacity one of these years.

CD is to blame. They are the parent company for TwinSprires which purchased the AmericaTab platform before Derby Day last year when they recognized that their own internal development effort to build an ADW platform had failed. Occasionally there are tote bottlenecks that cause problems, however there is not reason to believe that happened today. Most likely the AmericaTab platform (which is over 10 years old) was never written to scale to todays usage levels and needs an overhaul itself. But TwinSpires doesn't OWN AmericaTab -- they are an affiliate of AmericaTab. TwinSpires took over the BrisBet/TSNBet affiliates, but there are other AmericaTab affiliates that have nothing to do with TwinSpires (like BetPad.com, where they moved my account when they took over). TwinSpires.com is just a front page that sends you to the AmericaTab servers for the real stuff. (Although I notice they don't use AmericaTab domain names anymore on the affiliates -- maybe they are rerouting from one server to another causing additional bottlenecks?)

I believe Beulah Park in Ohio is the owner/manager of AmericaTab, correct? Issues of capacity, etc, would be up to them...

[DJofSD]

Let me throw two more things into the mix.

First, CDI and TS are attempting to expand their market. Don't you think they would have had a plan to increase capacity of their IT resources in anticipation of the additional activity, especially on a day like today?

Second, do you think the different stock exchanges have a problem planning for spikes?

[NoCal Boy]

Did the Youbet and/or TVG platforms just become more valuable after today?

[rcknhrse]

My wife advised me to go to charlestown today.But i am still recovering from knee surgery and that is a long drive.
Rosecroft is such a dump that was out of the question.
The derby appeared a difficult race to figure.
So i was taking in everything and just sifting the info thru my noggin.
I dont remeber exactly when i tried to get on but i am sure it was more than a half hour before post.
From now on i guess i have to make bets early on the Big days .
DERBY,PREAKNESS BELMONT AND BREEDERS CUP.
Not getting my bets down really cost me.
I cant sleep.I just took 2 vicadin and they are not helping.
This horse racing thing is an addiction i believe ,for me any how.
And getting the bet in is the fix.Win or lose ,i needed that release.
I f i had gotten the bets in and lost ,i could deal with it alot more then not getting them in at all

[richrosa]

GameTheory,

I gave out my opinion and stand by it. I professed professional expertise in this area. This is what I do for a living.



But TwinSpires doesn't OWN AmericaTab -- they are an affiliate of AmericaTab. TwinSpires took over the BrisBet/TSNBet affiliates, but there are other AmericaTab affiliates that have nothing to do with TwinSpires (like BetPad.com, where they moved my account when they took over). TwinSpires.com is just a front page that sends you to the AmericaTab servers for the real stuff. (Although I notice they don't use AmericaTab domain names anymore on the affiliates -- maybe they are rerouting from one server to another causing additional bottlenecks?)

I believe Beulah Park in Ohio is the owner/manager of AmericaTab, correct? Issues of capacity, etc, would be up to them...

Please refer to this press release last year where Churchill acquires AmericaTab. They are the bozos responsible for not caring and feeding their platform by doing to proper exercise of capacity planning and architecture management.

YouBet has done better but not much better. I'm not aware of any substantial problems with the TVG platform.

[DJofSD]

Debt Resolve, Inc.?

[richrosa]

yup. That's a company we founded in financial services selling web platforms to credit card companies. Scalable.

I cut my teeth on load testing being responsible for technical operations for 1800Flowers.com. I ran all of the websites worldwide for Scholastic Corp., the bookseller. Flowers and Harry Potter generate 10x more traffic than Derby Day for Twin Spires.

[DJofSD]

So, why hasn't CDI (or whomever) at least hired you as a consultant <g>?

BTW, I just noticed, over $22 million in the exacta pool on the Derby. Just the track take on that one pool alone would have paid for a lot of hardware.

[richrosa]

So, why hasn't CDI (or whomever) at least hired you as a consultant <g>?

BTW, I just noticed, over $22 million in the exacta pool on the Derby. Just the track take on that one pool alone would have paid for a lot of hardware.

What I do is not unique. They should have at least one like me at their place. I mostly consult now. They could hire me. I'd fix things there.

[PaceAdvantage]

With TwinSpires (and other AmericaTab affiliates) being one of the few (if not the ONLY) place to bet the Derby online (YouBet did NOT offer the Derby, and I don't think TVG did either...perhaps XpressBet did, but what do they have, 10 registered users?), is it any wonder their servers took more of a hit than usual?

[GameTheory]

GameTheory,

I gave out my opinion and stand by it. I professed professional expertise in this area. This is what I do for a living.Good for you. I'm not challenging your expertise. You seem to think I am.

Please refer to this press release last year where Churchill acquires AmericaTab. They are the bozos responsible for not caring and feeding their platform by doing to proper exercise of capacity planning and architecture management. Correct, thank you. I thought they had just taken over some of the affiliates but they are now in fact the managing partner.

[HEY DUDE]

You can hackle this thing to death. However, one thing remains. I am the customer and I really dont care what your problems are, just dont make them mine. This is very simple.

In the 4 years that I have been dealing with TwinSpires (AmericanTab), this has never happened.

[mikesal57]

i was locked out too...but good thing I had a backup site...nyra.com..but theres no live feed...

[shanta]

Did the Youbet and/or TVG platforms just become more valuable after today?

The "Al Capone" old school "platforms" certainly did man. ;)

[DJofSD]

They could hire me. I'd fix things there.

Yes, that would be fun. But then I'm know for having a strange sense of fun like volunteering for D/R tests.

[trigger]

Did the Youbet and/or TVG platforms just become more valuable after today?

Don't think Youbet's United Tote platform is more valuable:

"On the day of the 134th running of the Kentucky Derby, the largest wagering day of the year for OTB, betting machines throughout the region went down for more than 60 minutes, leaving frustrated betting fans fuming in lines and lounges.
>>>>>>>>>>According to Capital Region OTB President John Signor, the malfunction was due to a glitch in computers with the organization’s tote company, United Tote. Capital Region OTB contracts with United Tote to process the bets and transmit them to pools at tracks, Signor said.
Signor could not estimate how much revenue was lost because of the malfunction.
>>>>>>>>>>>Bettors at several locations pointed out that the period leading up to post time is typically when odds are refined and racing fans will place their bets. A number of disgusted fans at various locations walked away while more casual fans arrived but didn’t bother to stand in line.
>>>>>>>>>Signor said United Tote contracts with Capital Region OTB to serve a region from Columbia County to the Canadian border and over to Madison and Oneida counties. The company also contracts with Vernon Downs Harness Track in Oneida, the Tioga Downs Harness Racetrack and the Finger Lakes Race Track, all of which were affected Saturday, Signor said.
>>>>>>>>>>>At Imperial Racing Center in downtown Schenectady, customers waited 20 minutes after the conclusion of the race for vouchers that were stuck in three of the four self-service betting machines when the malfunction occurred."
http://www.dailygazette.com/news/2008/may/04/0504_otbgoesdown/

[NoCal Boy]

LOL...at least UT worked well at Churchill, Keeneland and NY. I could imagine the angry patrons in Albany.

[rrpic6]

Once you are at your limit I don't think there is a way to tell how much traffic you are rejecting. Let's say they are right this minute trying to figure that out for next year -- I'm not sure they can tell how much more capacity they needed -- only that they needed more. (BTW, this is an AmericaTab issue, not TwinSpires per se. TwinSpires is just one affiliate of AmericaTab. So CD is not to blame here unless the host track itself is causing the bottleneck -- possible I suppose.) From their point of view, everything probably looked fine -- they took all the bets they could take. There didn't seem to be any actual crash or downtime. (I managed to get in some late bets, but had to try several times.)

Not true..Twinspires bought out AmericaTab but did not keep many of the Winticket/Americatab personnel. Lots more chiefs and not enough Indians at CDI/Twinspires
RR

RR

[DJofSD]

According to Capital Region OTB President John Signor, the malfunction was due to a glitch in computers with the organization’s tote company, United Tote. Capital Region OTB contracts with United Tote to process the bets and transmit them to pools at tracks, Signor said.

John Signor has offered us absolutely nothing. Telling us there was a problem with a computer is about the same as a mechanic telling a customer your car had a problem and we fixed it. If your repair tech told you that, would you be satisfied, pay the bill and leave? I certainly wouldn't. The consistent lack of details about these failures does not serve the customers.

[Fredo515]

I kept trying and trying and finally got the bet in and won !
I would have been pretty angry if I didn't get it in--I had just about given up when it let me make the bet.

No need for that !

[whobet]

It didn't crash. It just became really slow because it was overloaded. All the betting sites seem to suffer this on every big racing day. It may not even be fixable -- they can't control all the networks and routers after all, only their own servers. More likely is that they could fix it by distributing lots of servers in different locations, but the expense would be so great to upgrade the capacity that it isn't worth it for 2 or 3 days a year when it is a problem. Or maybe they have upgraded and it just wasn't enough. There is also really no way to tell how much capacity you need unless you already have too much. And they won't be able to find that out until next Derby Day, although Breeder's Cup or the Belmont if you've got a possible TC winner are also very big.

Once you are at your limit I don't think there is a way to tell how much traffic you are rejecting. Let's say they are right this minute trying to figure that out for next year -- I'm not sure they can tell how much more capacity they needed -- only that they needed more. (BTW, this is an AmericaTab issue, not TwinSpires per se. TwinSpires is just one affiliate of AmericaTab. So CD is not to blame here unless the host track itself is causing the bottleneck -- possible I suppose.) From their point of view, everything probably looked fine -- they took all the bets they could take. There didn't seem to be any actual crash or downtime. (I managed to get in some late bets, but had to try several times.)


It is certainly fixable,

I am a IT professional, there is no excuse for this happening,
The other 364 days of the year does not generate anywhere near the LOAD KY DERBY does, they are losing handle cause they are Stupid by being cheap thinking if there site works for 364/365 days then that is acceptable, wrong.

How long has this been going on,
I am usualy at the track and never noticed before.

[JustRalph]

I have used Winticket/Ameritab for the last 6 years and only one year prior to this one did it go down. 2003 if I remember, And that only lasted about five minutes.

[NoCal Boy]

Comparing the performance of AmericaTab's platforms now verus past years is useless as the past years had TVG and Youbet taking the vast majority of handle on major race days. ATab's platform likely never experienced the type of handle volumes generated on Saturday.

[Spectacular Sid]

Here's a sorry-azz statement from Churchill's dim bulb VP technology (if largest CDI shareholder Dick Duchossois knew they screwed up this badly, Niven would already be shown the door). I'm sure this statement came out only after the press starting asking questions. It was not included in their original recap of business on Derby Day, which blamed the horsemen at Calder, Kentucky and Lone Star for lower handle.

Losers.

VERNON NIVEN STATEMENT REGARDING TWINSPIRES.COM WAGERING DIFFICULTIES ON KENTUCKY DERBY DAY

MOUNTAINVIEW, Calif. (May 4, 2008) – Vernon Niven, president of TwinSpires.com today commented on difficulties experienced by some TwinSpires.com customers placing wagers on the Kentucky Derby Presented by Yum! Brands on Saturday, May 3.

“We understand that some of our players experienced issues logging in and making wagers through TwinSpires.com during the 45 minutes leading up to the Kentucky Derby,” Niven said. “We are looking into what caused those problems and hope to have a definitive answer very soon. We deeply apologize for what we consider to be an unacceptable occurrence on the biggest racing day of the year. Our players deserve the very best, and we are working to make sure whatever issues caused them to have difficulties on Derby Day never happen again.”

[Tex9Down]

Tom said he wrote his congressman for government oversite.......


Oh no, please don't get the government involved. That would be the end of this sport for sure. :bang: