Is there any need for an open source data processing tool?

[BenDiesel26]

You will see that I mentioned Python and some of it's packages in my previous post. I am highly proficient in Python as well as R. The problem is, at this point Python's statistical and data analysis capabilities lag behind R's in my experience. "pandas" with numpy, scipy, and matplot lib is a good start, but it's still not there. There are some speed advantages in some areas, but it depends on what you are doing. For example, both are painfully slow in running loops, which is alleviated by R's sapply-type functions (and some similar stuff in pandas). I have also recently found that R's optim() function as well as the numerical integration routines outperform scipy.minimize and scipy.integrate for a particular maximum likelihood problem I am working on by a significant amount. Obviously with optimization, the speed is problem dependent. But nevertheless it surprised me.

Unless you need to to write a graphical user interface, the capabilities of R and Python are pretty similar. If you are just taking data files and manipulating the data, my point is you can do that right this second in R with zero overhead. And to the mention of C/C++, you can write routines in these languages (and Fortran) and easily call them in R. In fact, that is what most of the R packages do, otherwise they would be ridiculously slow if they relied on calling R functions.

Ultimately if your goal is to manipulate a large database of data and you are looking to reduce the code writing time as much as possible, R just simply can't be beat with Python a close second. If on the otherhand you are building a financial-type model that needs to be calibrated frequently (possibly up to the minute) to remain effective using a huge database, then you will want to go the C/C++ route.

[TonyMLake]

Actually, when I wrote that, I was thoughtlessly mixing my paradigms and didn't care what flavor of C.

In fact, at the time i was leaning towards C because of its universality.

After thinking about it though, now I'm leaning toward c++ or even java.

I don't know enough about python to comment on it, but it should be trivial to write the statistical routines, and by going with c++ or java you're going to have a lot more developers available to contribute, I'd think.

IMHO, that is.

[DeltaLover]

I have pushed the code to calculate any universe of handicapping factor under the following repo:


Location of Handicapping factor processor
[https://github.com/deltalover/thogar/tree/master/src

The mysql schema can be found under the same github repo while the factors are defined here:

Recency handicapping factors
[https://github.com/deltalover/thogar...apping_factors

The output is created by the program factoranalysis.py

Here you can see a sample output of the program
[http://themindofagambler.com/recency-analysis.xlsx


The existing handicapping factor universe specializes on recency, using all the possible increaments between 35 and 185 days as layout.

The next program I will check in will use this code to find the optimal layouts in such a way to maximize the crowd's error.

See here also for subsequent bug fixes and related info.
http://themindofagambler.com/mediawi...tor_Calculator

[traynor]

Any comments from anyone who has downloaded and is using this package? Good, bad, I dunno yet?

[DeltaLover]

Originally Posted by traynor

Any comments from anyone who has downloaded and is using this package? Good, bad, I dunno yet?

If anyone needs any help about how to install and run these programs, please let me know.. It should be straight forward, assuming a basic understanding of python, mysql , mongodb, vagrant and git..