Finance, Scientific Users Get ActivePython Updates 131
jcasman sends along this clip from PCWorld: "ActiveState has added three open source mathematics libraries to its ActivePython Python distribution that might interest financial and scientific computing markets, the company announced Thursday. The packages are being added, in part, to anticipate the demand that may arise from new proposed rules for the US financial community brought about by the US Securities and Exchange Commission. ... In April, the government agency posted a set of proposed rules for handling asset-backed securities that called for financial firms to disclose, along with their prospectus filings, the source code of the programs that generated the filings, as rendered in Python. The government agency will be accepting input about the proposed rule until August 2. The three libraries that are being added to the ActivePython package are NumPy, SciPy, and matplotlib."
advanced financial modeling (Score:1)
IOW fudging the numbers.. only faster and easier
Re: (Score:1, Funny)
only faster and easier
Since when has python ever been faster at something? *ducks*
Re: (Score:1, Insightful)
Re: (Score:2)
I am a C# programmer and I assure you I am the biggest dick.
We'll consider this matter settled, then.
Re: (Score:2)
Seriously, the performance gap between a hotspot optimized java implementation and python is generally a difference of about 20 fold for crunching numbers. For python to equal java in performance, the startup time of the vm has to represent the bulk of the work to be done. And then performance does not really matter that much because the size of your dataset is small enough it won't matter much.
Re: (Score:2)
Re: (Score:2)
Being written in c doesn't make it inherently faster than java. I've seen studies that have shown java running between 15% faster than c and less than 100% slower than c for specific algorithms.
It could go either way. Someone would have to do an actual comparison to see how these libraries stack up.
Re: (Score:1)
Re: (Score:2)
You are wrong. I am not going to say that I know much about optimization and performance of Python using c libraries, I don't. I do know that interpreted python is many many times slower than java. I do however know a lot about high performance java. Most of the time, it is comparable to writing in c/c++, a few cases run faster, most run slightly slower. One specific instance where java dramatically outperforms c is in allocation of memory on the heap, java does this in ~60 instructions vs c++ doing it in ~
Python is faster (Score:2)
Because in the time that I save writing the program in Python rather than Java, I can run my program a million times. And even if I run it a million more times, computer time is nearly free, so it does not matter. And when it really does matter, in less time than it takes to write in Java, I can rewrite the single time consuming routine in C, and exceed Java's performance.
Re: (Score:2)
If you can run the program a million times, it was trivial to begin with and you might have been faster just performing the task manually ;)
Re: (Score:1, Interesting)
I don't know how it's moderated Flamebait. As a matter of fact Python is faster at number crunching if you use the appriate libraries (scipy,numpy). These are highly optimized C libraries, and they don't treat numbers as objects. (And you don't use the builtin operators on them either.) And as a matter of fact when Python calls C libs, it has less an overhead than when Java does it. (However, by now Java has some similar scientific packages as well, but they aren't as high profile as numpy and scipy.)
I can'
Re: (Score:2)
Let me preface my comment with if the tool is fast enough it should not matter that a different tool is faster...
Python isn't faster than Java for number crunching - that's C. Python is just the glue. I could say the same thing for Java if JNI was being used.
Pure Java beats pure
Re: (Score:2, Insightful)
From http://www.sec.gov/rules/proposed/2010/33-9117.pdf [sec.gov] (p 205-206)
This proposed requirement is designed to make it easier for an investor to conduct a thorough investment analysis of the ABS offering at the time of its initial investment decision. In addition, an investor may monitor ongoing performance of purchased ABS by updating its investment analysis from time to time to reflect updated asset performance.338 In this way, market participants would be able to conduct their own evaluations of ABS and may be less dependent on the analysis of third parties such as credit rating agencies.
The waterfall is a critical component of an ABS. Currently investors receive only a textual description of this information in the prospectus, which may make it difficult for them to perform a rigorous quantitative analysis of the ABS.339
Cool job! (Score:2)
Re: (Score:2)
It does not really matter. You can still execute it and do a lot of useful analysis. Simply sweeping over the input space and plotting the output will help. If the output is really crazy (chaotic, whatever), then you can conclude that the asset is too complex and risky to buy.
Re: (Score:2)
What the hell is ActivePython and why the hell would anyone use it?
I've been using Python, numpy, scipy and matplotlib, and I know of absolutely no issue on any platform. How does this "news" affect me (or anyone)?
Re: (Score:1)
Although now available for multiple platforms, originally ActivePerl/Python/Tcl's main attraction is that they were a commercial port of popular UNIX scripting languages to Windows with the added feature of being able to interact with ActiveX components. This made them attractive alternatives to VBScript.
Re: (Score:2)
ActivePython is just another CPython Distribution. Previously, the SciPy and NumPy libraries were modules were only found 'pre-compiled' in the Enthough Python Distribution. Now when I download ActivePython I have two fewer modules to hunt down. In the windows world, it is a pain to try and build scipy and numpy. It's not enough to have the correct version of C and C++, you need a Fortran compiler as well.
activestate.com (Score:2)
here [activestate.com]
Hmm.
ActiveState has left a bad taste in my mouth in the past. My quick research just now may have dug up some reasons to re-evaluate them.
Re:Great! (Score:4, Insightful)
That link is unrelated to this post. The referenced scientific/numeric libraries for python are implemented in C as native modules. They are not just as fast as the equivalent code written in C, they *ARE* the equivalent code written in C, merely interfaced to with python. You might lose a meaningless tiny fraction of time preparing your vectors for processing in python, but you'll save several orders of magnitude more time not worrying so much about malloc corruption.
Re: (Score:3, Informative)
I don't think you understand. These things are used as any other python code. The magic is that C is the backend that actually crunches the numbers. You don't even have to know or care that C is used. All you know is you call the foo method in the bar module.
Re: (Score:1)
You don't even have to know or care that C is used.
Well, unless you're not using CPython.
Re: (Score:3, Informative)
It is only irrelevant if all the computations you are performing are done by the libraries. I tried using python for data processing tasks, and it was unbearably slow despite the use of scipy. I think it was due in large part to poor I/O and bit-twiddling performance while reading/writing data files but I'm not sure. Anyway with the amount of time I spent optimizing code, I could have just written the damn thing in C to begin with. I just don't understand python's poor performance. All it's high level langu
Re: (Score:1)
Python is actually more dynamic than most Lisp object systems...
Did you use Python 3.0? The IO performance there was a big drop from 2.x and was largely fixed in 3.1.
Re: (Score:2, Interesting)
Re: (Score:2)
Just use Cython.
All the niceness of Python and the speed of C where you need it.
Re: (Score:2)
Which might be great if you need to do a one-off calculation. If it takes you 5 minutes instead of 10 mins and runs 10 seconds instead of 1 second then you're saving 4 minutes 51 seconds :)
Re:Great! (Score:4, Interesting)
For about an average of half the lines of code they might use in C, scientists ...
Scientists?! You're probably joking but I've been over to accounting and they're using Excel and *shudder* Access for all their heavy data lifting. Sometimes they need help and if they're kind enough I don't lie about how much I know about those ancient products. Most importantly they're not scientists. They're accountants and business people ... they don't care if they have to wait five minutes for Excel to open a worksheet containing the entire set of order histories of our company.
...
I don't think these packages are intended for NASA space mission flight certified calculations. Just something to really help you out if you want to comply with the SEC. Side bonus, I'll bet that when you submit this code, you're going to achieve compliance a lot faster when the SEC only has to check half the lines of code and not analyze your memory management
Software development is about trade-offs. Why aren't you complaining that it's not in some targeted processor specific machine language?
Re: (Score:3, Interesting)
I know bonafide scientists who use Excel for analysis. Scary.
I also know quite a few who use numpy, scipy, matplotlib, and python for real science. In fact, I think the astro community specifically was an early supporter of some of these packages. I myself have been using them in science for 5 or 6 years.
Re:Great! (Score:5, Interesting)
Re: (Score:2)
For many applications, especially quick one-time tasks, coding takes a lot longer than execution. If you can cut the coding time in half from 60 minutes to 30 minutes, does it really matter if the execution time goes up from 5s to 100s?
Re: (Score:2)
Re: (Score:2)
This is not some sort of high-frequency trading or other section of finance where speed matters. This is producing SEC filings...As long as execution time is not in the month+ range, you are going to be fine.
Great tools (Score:2, Insightful)
Does Not Work with Most IDEs! (Score:5, Funny)
Oh and I also tried double clicking the packages and nothing happens
Will someone please rewrite this in Visual Basic so I can do my job?
Re: (Score:1)
Re: (Score:3, Informative)
You need to either write COM component in Python and interface with Visual Basic that way, or use IronPython and call it from .Net like this [msmvps.com].
Re: (Score:2)
The shitty end of the stick finally comes around and slaps em in the face. Bought into Microsoft? Sure it was painless. At first. It takes a good while to set in, but like a burn you feel the fucker nice and good.
Re: (Score:1)
Re: (Score:1, Funny)
>> I'm a pretty talented hacker (MSCD and everything) and this just goes to show how crappy open source can be. :D
You need to replace Visual with Vim, Microsoft with a Unix flavour, drag-and-drop with shell, unzipped with untarred or unbzipped2, WIN32 with BIN, Basic with Python or C before you will be recognised here as a hacker
Re: (Score:2)
unbzipped2
Isn't that bunzip2?
Re: (Score:2)
Just goes to show the value of a MSCD now doesn't it. You may wish to start with a little trip through the Win32 API.
Does Not Work with MSFT IDE! (Score:1)
Re: (Score:2)
The funny thing is, anyone who can type "easy_install" on the command line doesn't really need Activestate's distribution and has been happily using Numpy, Scipy and Matlotlib for years.
In some ways, ActivePython is like the VB of the Python world.
Termination? (Score:1)
While we're on the topic, here's something that confused me about the SEC / Python idea.
Python is Turing complete, which means some Python programs may never terminate*. Has the SEC taken this into consideration in its plans to use them?
Or is the SEC planning to impose limitations such as, "These Python programs must complete within 1 hour when run on an Intel Pentiun IV 2.8 GHz with 4 GB RAM and Windows XP SP 3"?
(* Of course, real computers have finite memories, so it's actually theoretically possible to
Re: (Score:1)
It seems fairly likely that the implementation of the regulation would require the models to be useful.
But maybe not.
Re: (Score:3, Informative)
Oops - that's embarassing. I just realized that the Python program must be those used to produce the actual filings. So the programs' fitness to purpose must have been already established. Presumably they don't loop forever, or at least only do so after producing the filings.
Re: (Score:3, Funny)
Or the loop was part of the design.
I leave my machine on overnight and lots of things are looping "endlessly" - and that's not a problem.
Re: (Score:2)
While we're on the topic, here's something that confused me about the SEC / Python idea.
Python is Turing complete, which means some Python programs may never terminate*. Has the SEC taken this into consideration in its plans to use them?
Or is the SEC planning to impose limitations such as, "These Python programs must complete within 1 hour when run on an Intel Pentiun IV 2.8 GHz with 4 GB RAM and Windows XP SP 3"?
(* Of course, real computers have finite memories, so it's actually theoretically possible to detect looping on such a computer. But at this point we get back to specifying a particular memory size I think, which kind of goes to my question about the SEC specifying the particular hardware on which the program must run.)
Gag, I hate Python; it has to be the most annoying language to code in since COBOL. White-space blocked languages needed to die with the Hollerith card.
Re: (Score:2)
If you think Python's syntax has anything to do with COBOL or Hollerith cards, I doubt you've ever even looked at any Python code.
Re: (Score:1)
Probably because Python is a more widespread skill than R. Python code is also extremely easy to read and understand to most average coders, even if they have little or no experience actually coding Python.
Re:Why not R? (Score:5, Interesting)
Speaking as someone who uses both R and Python all the time, I'd say that while R is very very good at a lot of what it does, it's just not as good as a general-purpose language as Python. I find myself doing as much preprocessing in Python as possible, then saving the results in DB tables and having R finish the analysis. And yes, I know about RPy, but the programming overhead of representing data structures in both languages, and making sure that they're talking to each other correctly, can be considerable; so is the runtime overhead of passing really big data sets back and forth. (Note that it's been a while since I used RPy for anything big, and a lot has been improved in that time, so it may be time for me to give it another shot.) Python code is just cleaner and easier to write for most tasks. I like both languages for their strengths, but overall, if you can do a particular analysis in Python then that's usually the easier choice.
Re:Why not R? (Score:4, Interesting)
Mod parent up!
I also use both, quite heavily. Rpy2 is a real improvement over RPy, but I share your preference for allocating tasks to the two languages according to their strengths, and the rpy/rpy2 data structure representations are much trickier than I like.
I find myself gravitating to R when the builtins are useful (robust estimation, smoothing kernels), where I need to split and analyze data subsets (using by() and its cousins) or where I want to plot things. Python is the ticket for interfacing with C, talking to the outside world, parsing, and expression of simple mathematical models.
Re: (Score:1)
Re: (Score:2)
I do use Sage for some work, but I didn't know about the RPy interface or the ability to run worksheets in R. That sounds well worth checking out. Can you point me to the documentation?
Re: (Score:2)
But I don't have much experience with R. How does it perform when you aren't using (and combining) the builtin methods, but are writing some number crunching routines of your own?
It's not bad. If you try to write naive C-style code, you'll run into all the overhead of an interpreted language and it will run very slowly. But if you (a) try to do everything you possibly can as matrix operations and (b) make intelligent use of the functional programming features, it compares well to NumPy/SciPy. And as with Python, it's easy enough to write your own packages using C or Fortran and then call them from the main language (which is how NumPy/SciPy works, of course.)
Good stuff (Score:4, Interesting)
Those packages are fantastic and really 90% of what I use in python are in those packages. I have been using enthought edition python rather than active-state (many reasons), and this tips the scales a bit more toward recommending active-state to others.
FYI: Matplotlib makes 2D and 3D presentation quality plots of data (even an absurd quantity of data). Numpy and scipy provide scientific and matrix functions that pretty much cut matlab off at the knees unless you are a simulink user. Matlab is many thousands of dollars, python is free, and they are both remarkable similar, except matlab chokes on large data sets where python doesn't.
Sheldon
Re: (Score:2)
Good points.
See Active State product description page and RTFA where the versions of Active state that will include this package are listed.
"ActivePython now includes NumPy,SciPy, and matplotlib in ActivePython Business, Enterprise, and OEM Editions"
http://www.activestate.com/blog/2010/06/saving-world-economic-collapse-python [activestate.com]
Product comparison:
http://www.activestate.com/compare-editions [activestate.com]
IOW, these packages are available for the lowest price possible (the Biz edition) of $750 per server/yr).
Re: (Score:3, Informative)
I use those packages on Windows without ActiveState... they are free downloads from their respective websites.
http://www.scipy.org/Download --- NumPy and SciPy
http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-0.99.3/ --- MatPlotLib
http://www.python.org/download/ --- Python
Re: (Score:2)
What he said! I love numpy+scipy+matplotlib. Makes my life soooo much easier.
Re: (Score:3, Insightful)
I would be very interested in what matlab is choking on that Python wouldn't. The only data I've had matlab give me errors on is stuff that starts breaking 32-bit memory addressing. (In 32-bit Windows).
I'm a hardcore matlab user and while there is no chance in hell my company would ever give it up for python, I'd consider it for some of my personal projects.
And as limited as it is, there is one feature of the Matlab Editor that I don't think my lazy self could live without. Ctrl-A Ctrl-I, select all and aut
Re: (Score:3, Funny)
Auto-indenting for Python? Are you serious?
Re: (Score:3, Insightful)
Yes, I am. I know absolutely nothing about Python. I earn my keep with Matlab. I know C (which is quite useful for CANape scripts) and use PHP in my free time.
I just asked a simple question. Rather than answering, you respond with a jackass comment of "ha, are you serious?". Which now just makes me want to disregard Python and all this crap and I'll just stick with my Matlab and short PHP script for doing repetitive stuff.
It's a little less forward than RTFM but just as condescending. And you wonder why com
Re: (Score:1)
Re: (Score:2, Informative)
Re:Good stuff (Score:5, Informative)
Grandparent thought you were joking because one of the special features of python is that it doesn't use brackets for statement grouping and instead uses indenting. Thus all working python programs must be properly indented. I guess you could still complain if someone doesn't use the number of indent spaces you like for each block, but I assume your big issue is the confusion of various indenting styles rather than just the size of the indent.
Re: (Score:2)
"Is there any python IDE with this built in... anything that mimics the 'desktop' of Matlab?"
TextMate. If you're not on a Mac you have a choice of all the other Python editors, one of which must surely have those basic functions.
Re: (Score:2, Informative)
Is there any python IDE with this built in... anything that mimics the 'desktop' of Matlab?
Why not have a look at Geany [geany.org]? It's small, free and sweet (i.e. very usable), and it assists with editing code in a lot of programming languages.
By the way, the key sequence you gave as example nicely increases indent one level in Geany. For Python, Geany will auto-indent some obvious things: indent after a line ending with ':', dedent after 'return', etc.
Re: (Score:1)
What do you mean by "list"? You mean echo to the screen? Print it out? Something else? Are you trying to say that you can't see the indentation? Really? If you can't follow 4-space indents (the recommended amount) visually then you are unlikely to be able to read the text accurately either.
It sounds like you prefer "line noise" languages that suffer from readability problems when abused in such fashion. More power to you. But you only sound confused and backwards when you can't even express your thoughts cl
Re: (Score:2)
you can't unambiguously list it, because on paper you can't tell the difference between a space and a tab.
You do realize that you can use spaces instead of tabs? You just have to be consistent in the way you indent. Your comment is invalid.
Re: (Score:1)
The OP is entirely correct. Python listings are brittle when you think about all the ways that they might be shared around: on HTML pages, on paper printouts, quoted inside a Word/OO document, quoted inside a LaTeX document, written in an email to a colleague, etc.
A piece of Python source code is not much better than a binary blob, when it comes to listing/communicatin
Re: (Score:2, Informative)
but while I was programming an important app, I accidently hit the space bar just before tabbing. Since this error wasn't visible on printouts or screen views, [...]
What, the glaring "IndentationError" exception that gets thrown as soon as you import the file didn't tip you off?
The situation you describe can never happen silently. I call bull$!7. Theoretically, it is possible to construct a situation where you would get a silent inconsistency, but this isn't it. I've programmed extensively in Python since '97 and never experienced problems due to indentation. In real life, this just isn't a problem.
But you can't blame them, I mean what sort of idiot language has whitespace signify blocks of code?
Well.... all of them do. How else would you find the blocks in a prog
Re: (Score:1)
I accidently hit the space bar just before tabbing. Since this error wasn't visible on printouts or screen views,
Get a proper editor?
Is this an Active Python Advertisement. (Score:1)
Not free, however (Score:5, Informative)
TFA and TFS fail to mention that SciPy, Numpy and Matplotlib have been added only to the Business, Enterprise, and OEM Editions of ActivePython. The Community Edition (the only one that's free) doesn't contain these libraries.
http://www.activestate.com/activepython [activestate.com]
Re: (Score:1)
wow. these people pay to install a python distribution? check the umpteenth proof that the economy went bad because stupid people are in charge.
seriously now... I use all these, and they're free, and in ubuntu I just had to click install to get them.
Re: (Score:3, Informative)
And what, exactly, prevents you from installing SciPy, Numpy, and Matplotlib into an already existing ActivePython community edition installation?
Hint: Nothing. You download the libraries and install with the canonical 'setup.py install'
Re: (Score:3, Informative)
Thus, they are free.
But if you want to fund python related staff, then fine.
Time to abandon Matalb? (Score:2)
Does this mean one less reason (scientific field) to use commercial Matlab and prefer free Sage/ActionPython/NumPy/SciPy/matplotlib?
Just asking, but since Sage can offer so much functionality, I wonder if now the community gets one more extra boost.
Re: (Score:2)
Does this mean one less reason (scientific field) to use commercial Matlab and prefer free Sage/ActionPython/NumPy/SciPy/matplotlib?
I'm not sure about ActionPython -- why pay for a bundle of free components? But I for one am a long-time Matlab user, and I am slowly migrating towards Python (using numpy, scipy, matplotlib, ipython, etc.). Migration is slow mainly because I have to recode some of my own Matlab tools, but so far it's been succesful. Not painless mind you, as the differences in numpy and Matlab can cause really subtle bugs.
So while it's great that ActivePython now gets numpy and matplotlib, I wouldn't say there is one less
Re: (Score:2)
That's one reply I was looking for. Indeed I agree that the situation is as you describe it.
Nevertheless, for the frugal scientist, Sage and there rest of Python toolkits seem like nice alternatives.
Re: (Score:2)
Or you can use Octave which is about the same performance as python + SciPy/NumPy/matplotlib, but has the added benefit of being largely compatible with Matlab.
Re: (Score:2)
I've been using Python + SciPy / NumPy / matplotlib / IPython at work recently and it's awesome. I don't have a MATLAB license, so whereas some of my colleagues are using MATLAB, I've been doing my plotting and analysis work with just Open Source tools. I played with Octave (albeit the Cygwin port) and SciLab but in the end, the Python-based solution gave me access to a core programming language that's very neat and that I like, plus it's really fast and proficient at dealing with large datasets. It's no
For OS X users to get matplotlib & co. (Score:2)
And matplotlib is a gem. It's got a ridiculuous number of plot-styles so it's remarkably flexible - if you are into GIS, look at
Re: (Score:2)
Unless Macports supports Aqua now, don't do it.
Rather:
1. Install Python (available as a double click install from python.org).
2. Install Matplotlib (available as a double click install form matplotlib.sourceforge.net
2 (alternate). At the command line type easy_install matplotlib
My Math Prof used Excel 4.blah on Win3.1 for this (Score:2)
I hate to essentially troll, and I hate to burst your bubbles, but these math packages aren't really doing anything all that wonderanomous. The guy I learned numerical analysis from in college used to use Excel to do a lot of his numerical techniques - and used to do a lot of them on a TI-80. Numerical analysis is all about knowing a lot to write an efficient algorithm to get the answer.
I've done Q/R decomposition in VB6 (for a real honest to god client! for money!)
I'm glad these tools are around for peo
Re: (Score:1)
Except that NumPy will use LAPACK and BLAS for it's linear algebra making it far more efficient. Try a QR decomposition on a matrix of any significant size in VB, then do the same decomposition using LAPACK and you'll see a huge difference. As for numerical analysis being about writing efficient algorithms, sure, that's true, but why would you want to rewrite those algorithms when highly optimized versions come by default?
Disclaimer: Yes, I'm sure you could get VB to use LAPACK and BLAS but python will do i
Re: (Score:2)
You're comparing apples and cadillacs. Excel doesn't use LAPACK either, but Prof used to dig it because of it's profound recursive capabilties. I didn't write the Vb6 code from scratch - it was originally written in Fortran using LAPACK there, and for some reason the guy wanted it in VB6 (I guess so he could enter the parameters from a database using a form.) It was a simulation of some complexity and ran well enough for his purposes. The same calculations were being done, on ordinary PCs.
It's a distinc