Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
Programming

A C++ Library That Brings Legacy Fortran Codes To Supercomputers 157

gentryx writes "In scientific computing a huge pile of code is still written in Fortran. One reason for this is that codes often evolve over the course of decades and rewriting them from scratch is both risky and costly. While OpenMP and OpenACC are readily available for Fortran, only few tools support authors in porting their codes to MPI clusters, let alone supercomputers. A recent blog post details how LibGeoDecomp (Library for Geometric Decompostition codes), albeit written in C++, can be used to port such codes to state-of-the-art HPC systems. Source code modification is required, but mostly limited to restructuring into a new pattern of subroutines."
This discussion has been archived. No new comments can be posted.

A C++ Library That Brings Legacy Fortran Codes To Supercomputers

Comments Filter:
  • Code... (Score:4, Informative)

    by gigaherz ( 2653757 ) on Saturday September 21, 2013 @11:39AM (#44912119)
    ...like rice, is not countable. At least not since I learned the word.
    • It really does lower one's opinion towards the author. If I read TFAs, I wouldn't read this one.

      • Re: (Score:3, Insightful)

        I couldn't agree more. Although the word "codes" is usually a red flag not to bother reading any more in any article or question
        • by jeremyp ( 130771 )

          Not to mention the fact that the author has erased history (well, the summary implies the author has erased history - I haven't read TFA) because the Cray 1 had a vector processing unit and a specially designed compiler to make use of it, and the compiler was for Fortran. This was in 1978 when C++ didn't even exist.

        • Re:Code... (Score:4, Informative)

          by cas2000 ( 148703 ) on Saturday September 21, 2013 @07:47PM (#44914625)

          actually, "codes" is common usage amongst researchers and has been since at least the 1970s.

          most of them are not programmers or geeks or computer scientists, they're researchers or academics or post-grad students who happen to do a little programming or simply use someone else's "codes".

          it used to make me cringe every time i heard it when working with academics and researchers and on HPC clusters, but then i got used to it and stopped caring.

          and, really, they're not interested in a lecture or why it's a dumb usage of the word. they've got stuff they want to get done ("codes to run") and don't give a damn.

    • Author here. (Score:4, Informative)

      by gentryx ( 759438 ) * on Saturday September 21, 2013 @12:36PM (#44912479) Homepage Journal
      The IEEE and Los Alamos National Laboratory seem to have a different opinion [google.com] on this. And even the Oxford dictionary knows the use of codes [slashdot.org]. But surely those guys can't even spell gigahertz.
      • I guess I was mistaken in assuming the computer-science way of thinking about the concept of "code" (vs. "program", "algorithm", or "function", which are definitely discrete, countable things) should extended to other fields.

        As a side note, although it is true that my nickname was originally a misspelling of "gigahertz", I chose it while I was young, and as a non-native English speaker, my knowledge of the language was lacking. I have been perfectly aware of that fact for a long time, but I chose to maintai

        • by dfghjk ( 711126 )

          "I guess I was mistaken in assuming the computer-science way of thinking about the concept of "code" (vs. "program", "algorithm", or "function", which are definitely discrete, countable things) should extended to other fields."

          You were not mistaken, the world just grows more poorly educated. Yesterday's illiteracy is today's literacy. Making "code" synonymous with "program" and subsequently requiring a plural form serves no purpose, sounds stupid, and is stupid.

          No doubt others wish to "commentate" on the

          • I'd suggest you don't be so pious. I'm for protecting the language as much as anyone else but ultimately it evolves. I don't think this is really about the people employing numerical techniques in science becoming "more poorly educated"; I think it's about your field branching out and attracting new jargon and new uses for the old jargon. It's just what happens.

            As it happens, I've spent close to ten years in academia where we build "codes" (typically in Fortran -- 90 or more recent if you were lucky; 77 if

            • "Corn" means "grain" (grain is from the French for corn, coming from a common root). "Corn" is only wheat by virtue of wheat being the most common type of grain... a "default", if you will. Similar to how I call a mallard a "duck", because it is the only breed found commonly where I grew up. (Daffy and Donald bost confused me when I was a child...)
              • Calling corn 'wheat' a default is a good way of putting it - it was what I was trying to get across in more words. "Corn" is ultimately related to "Kernel" and does certainly come from "seed", "grain". But the language evolved so that now it's a catch-all for grains, and wheat by default across most of Europe. Of course, I want the kids off my lawn and the language to stop evolving ;)

          • I had some common taters for dinner.

          • You were not mistaken, the world just grows more poorly educated. Yesterday's illiteracy is today's literacy.

            Forsooth, good sirrah, thou hast spake sagely, and shewn thyself more wise that thy wyrd wurds should haue me think.

        • by gentryx ( 759438 ) *
          Just to add another twist: even as an English native speaker I would not be surprised if you spelled Hertz wrong since it's a German name, and because Herz and Hertz are pronounced identically in German, it's even a common misspelling in Germany, too. :-)
    • never worked in the field of high performance numerical methods?

      • Obviously not. Even after reading the other replies, and realizing that I was partially wrong, I still can't help but feel that it sounds wrong.

        The concept of code, to me, is a collection of statements, expressions, functions, packages, etc. Like a bowl of rice, it makes no sense to count the grains for themselves. I can accept that other people understand it differently, but it's not easy not to feel that they are doing it wrong...

        • by jythie ( 914043 )
          That is one of the things that happens when multiple disciplines overlap, sometimes their jargon does not always match up.
    • by Cito ( 1725214 )

      it is correct though

      similar to how moneys and monies are both correct plurals of money, even though in America people use money to refer to both singular and plural but they should recheck the dictionary.

      codes is interchangeable as well http://www.thefreedictionary.com/code [thefreedictionary.com]

    • I studied math in college, and many numerical algorithms textbooks refer to software as "codes". It seems to be common practice in the computational mathematics world. I assume it goes back to the days before Fortran, before high-level languages in general, when source code literally consisted of a series of codes.
    • ...like rice, is not countable. At least not since I learned the word.

      It's a shiboleth of physicists and lousy journalists.

    • by manicb ( 1633645 )

      "Secret codes" or "cipher codes" are countable. So are "weather simulation codes".

    • It's funny, that was my first response, before even starting the summary. The writer seems to know nothing of the difference between codes and code. Baby, bathwater, out.
  • by RoverDaddy ( 869116 ) on Saturday September 21, 2013 @11:51AM (#44912203) Homepage
    I took a look at TFA and followed up by reading the description of LibGeoDecomp:

    If your application iteratively updates elements or cells depending only on cells within a fixed neighborhood radius, then LibGeoDecomp may be just the tool you've been looking for to cut down execution times from hours and days to minutes.

    Gee, that seems like an extremely limited problem space, and doesn't measure up at all to the title of this Slashdot submission. It might really be a useful tool, but when I clicked to this article I expected to read about something much more general purpose, in terms of 'bringing Legacy Fortran to Supercomputers'.

    By the way, regarding the use of the word 'codes': I don't think English is the first language of this developer. Cut some slack.

    • AFAIK a lot of simulation problems are centered around 'update node based on neighbors', like particulate dispersal or flux.
    • FTA:

      "My idea for this post was to take an simple 3rd party program and try to marry it with LibGeoDecomp while preserving as much as possible of its original code and structure. The Hello World equivalent for computer simulations is Conway's Game of Life. My choice fell on this code, kindly provided by KTH, Sweden. The code has several advantages:"

      In regard to:

      "By the way, regarding the use of the word 'codes': I don't think English is the first language of this developer. Cut some slack."

      You really think t

      • You really think that sentence was written by a person with a tenuous grasp of the English language. Seriously?

        Tenuous grasp, no; but non_native != tenuous_grasp. The blog's at a German university. The choice of "marry" over "combine" is slightly unusual, as the idea of choice falling on something. It's very, very good. But it's still most likely not his first language, so pedantic polemics are uncalled for.

    • Very limited indeed (Score:5, Informative)

      by gentryx ( 759438 ) * on Saturday September 21, 2013 @01:03PM (#44912601) Homepage Journal

      I took a look at TFA and followed up by reading the description of LibGeoDecomp:

      If your application iteratively updates elements or cells depending only on cells within a fixed neighborhood radius, then LibGeoDecomp may be just the tool you've been looking for to cut down execution times from hours and days to minutes.

      Gee, that seems like an extremely limited problem space, and doesn't measure up at all to the title of this Slashdot submission. It might really be a useful tool, but when I clicked to this article I expected to read about something much more general purpose, in terms of 'bringing Legacy Fortran to Supercomputers'.

      Correct. We didn't try to come up with a solution for every (Fortran) program in the world. Because that would either take forever or the solution would suck in the end. Instead we tried to build something which is applicable to a certain class of applications which is important to us. So, what's in this class of iterative algorithms which can be limited to neighborhood access only?

      It's interesting that almost(!) all computer simulation codes fall in one of the categories above. And supercomputers are chiefly used for simulations.

      By the way, regarding the use of the word 'codes': I don't think English is the first language of this developer. Cut some slack.

      Thanks :-) You're correct, I'm from Germany. I learned my English in zeh interwebs.

  • I think I speak for many geeks when I say....

    KHHHAAAAAAAAAAAAAANNNNNN!!!!

    That is all.

  • Seems to me that there are bigger problems when porting Fortran code to C++, like lack of a multidimensional array type in C++, lack of all the other Fortran libraries, and the fact that Fortran code usually still seems to give faster executables than comparable C++ code on numerical applications.

    • by bored ( 40072 )

      fortran code usually still seems to give faster executables than comparable C++ code on numerical applications

      I don't think this is true anymore. C++ is pretty much the only language that has BLAS libraries that can actually beat the fortran ones. The latest C++ template libraries are using SSE/etc vector intrinsics and are capable of meeting if not exceeding the fortran performance for many applications.

      But, if you have a bunch of code in fortran, its probably not worth the trouble to convert it.

      • It's still true. Fortran uses these intrinsics as well, furthermore the way Fortran handles variables is stronger than C/C++, which permits the compiler to perform more aggressive optimizations. Fortran also has convenient syntax for performing common mathematical operations on datasets. Yes, you can replicate this in C++ with operator overloading, but Fortran puts this in at the core language permitting the compiler writers to target these specific operations for optimization.

        Lots of existing code is
      • C++ is pretty much the only language that has BLAS libraries that can actually beat the fortran ones.

        Why would any Fortran compiler be using a slower BLAS implementation than the C compiler?

        The latest C++ template libraries are using SSE/etc vector intrinsics and are capable of meeting if not exceeding the fortran performance for many applications

        Hand-tuned C code is "capable of meeting if not exceeding the fortran performance for many applications", but that doesn't make C a good numerical programming lang

    • We're using Boost Multi-array [boost.org] as a multi-dimensional array, so that's not really a problem. And since we call back the original Fortran code users are still free to use their original libraries (some restrictions apply -- not all of these libraries will be able to handle the scale of current supercomputers).

      Regarding the speed issue: yeah, that's nonsense today [ieee.org]. It all boils down writing C++ in a way that the compiler can understand the code well enough to vectorize it.

      • by emt377 ( 610337 )

        You never want a compiler to vectorize code. You want interfaces to vectoring hardware that you use to vectorize operations on your data. Just like you don't want compilers to provide multidimensional arrays - memory isn't multidimensional, so there's no natural layout. Instead you implement the arrays you need - even if they look the same the complexity contract and implementation is completely different for statically dimensioned (e.g. template params in C++) vs dynamically dimensioned (can be resized)

        • You never want a compiler to vectorize code.

          I most certainly do.

          Just like you don't want compilers to provide multidimensional arrays - memory isn't multidimensional, so there's no natural layout

          There is a natural layout that handles 99% of all numerical needs. Numerical programmers understand it, and so do compilers.

          NONE of this has a natural representation.

          You listed a bunch of exceptional cases that should indeed be handled by libraries. But not to support common cases well because of exceptional cases i

      • We're using Boost Multi-array [boost.org] as a multi-dimensional array

        Boost Multi-array doesn't support most modern Fortran array features, so it's useless for porting modern Fortran code to C++: you end up having to rewrite most of the code from scratch.

        Regarding the speed issue: yeah, that's nonsense today [ieee.org].

        That just shows that with enough effort, you can create efficient special purpose libraries in C++; of course you can. The question is whether straightforward, boring numerical code compiles

        • by gentryx ( 759438 ) *
          Care to backup those claims with actual code/numbers? I'm just asking because my FUD alarm just rang. Part of my job is performance engineering. My experience is that if you use C++ correctly, you get code which at least matches Fortran code.
          • Re:FUD (Score:4, Informative)

            by stenvar ( 2789879 ) on Saturday September 21, 2013 @05:47PM (#44914005)

            Care to backup those claims with actual code/numbers?

            You claim to be writing high performance code and you don't understand the difference between Boost multi-array and Fortran arrays? I'm sorry, but if you do any kind of high performance computing, you should at least have a decent understanding of one of the major tools used for it, namely modern Fortran. Once you do, you can then make an informed choice, instead of behaving like an immature language zealot.

            Here are two places you should start looking:

            http://en.wikipedia.org/wiki/Fortran_95_language_features#Arrays_2 [wikipedia.org]

            http://en.wikipedia.org/wiki/High_Performance_Fortran [wikipedia.org]

            (The Fortran code on libdecomp.org is cringe-inducing and inefficient.)

            And, FWIW, I'm primarily a C++ programmer, because that's what the market demands, not a Fortran programmer, but at least I know my tools and their limitations.

            My experience is that if you use C++ correctly, you get code which at least matches Fortran code.

            If you use C, assembly, or Java "correctly", you can usually match Fortran code. That is entirely not the point.

            • by gentryx ( 759438 ) *

              So you said Fortran codes we faster than C++ codes and now that's not the point any longer as they really aren't? Great, thanks!

              The links you provided show that Fortran has some convenience functions for selecting parts of arrays and applying arithmetics to them. What I didn't see is anything you can't so with Boost Multi-Array and Boost SIMD [meetingcpp.com].

              • So you said Fortran codes we faster than C++ codes

                I said no such thing; that doesn't make any sense. What I said is:

                Boost Multi-array doesn't support most modern Fortran array features, so it's useless for porting modern Fortran code to C++: you end up having to rewrite most of the code from scratch.

                and

                That just shows that with enough effort, you can create efficient special purpose libraries in C++; of course you can. The question is whether straightforward, boring numerical code compiles into fast executa

                • by gentryx ( 759438 ) *

                  Sorry, I got overexcited and did see something in your post that apparently wasn't there.

                  And yet I don't buy into this "OMG, C++ is either clumsy or slow compared to Fortran" FUD (I hope I'm paraphrasing it correctly this time). For a certain (perhaps smallish) domain LibGeoDecomp is such a library which makes it easy to write short, yet (nearly) optimal code with C++.

                  I don't doubt though that there are use cases where it's hard to come up with a good C++ solution while Fortran would outperform it in both,

    • If since 1979 a language hasn't manged to support things like multidimensional arrays maybe taking it out behind the back of the barn with a shotgun might be the best solution.
  • And why would you fuck about with C++ when there is so much missing - just get a book and learn FORTRAN if you need to work in the scientific computing environment.
    • by Greyfox ( 87712 )
      If I were doing new development, which I am, I would use C++ because like Java and Ruby I can develop code in it more quickly, make my libraries more user-friendly (Where the user in this case is another programmer or myself later on) and am more likely to be able to find other programmers who can use those libraries without requiring them to learn an ancient language that just barely qualifies as structured-programming-capable. And yes, I HAVE written fortran code. And assembly, back in the day.

      Unlike Ru

      • Then you're likely a waste of time and detriment to your team.

        I have had conversations with some of my friends who work on the peta-scale clusters and thought much the same as you. But, it turns out, when you're working with that level of system, you're probably addressing some small part of a much, much larger problem that has been largely solved. The existing code that performs 99.9% of your task is written in Fortran and actively developed by a very successful team of researchers. Attempting to rewr
        • by Greyfox ( 87712 )
          You didn't really read my post, did you? I'm perfectly fine being presented a complete system written entirely in FORTRAN and being told to support it. If I'm doing NEW DEVELOPMENT, I prefer C++.

          Thing about those old systems, they typically weren't written by dumbasses. Most of my career has been following along behind dumbasses cleaning up at them. It's lucrative work, and I'm never hurting for something to do. Every so often I happen upon a system that was actually written by engineers and it's usually

      • That's because you aren't doing development on computationally expensive simulation codes that run on supercomputers. Because then you would use FORTRAN. C++ is such a memory hog, and the memory overhead scales with the number of processors. In FORTRAN, you only allocate what you need to use, and that's important when working with large arrays. Java and Ruby are out of the question.

        FORTRAN is not obsolete, because there are currently no other languages that can fill the role. When running simulations that t

    • by jythie ( 914043 )
      Many technological decisions are based off how easy it is to find programmers to fill roles, which means if you want to be able to easily hire people onto a project you have to bend to what is generally popular. I worked on several projects that ended up switching languages or even OSes because what we were using made finding candidates more difficult.
      • yeah that is how I started at a world leading Rnd organization was told to get a book from company library and teach my self FORTRAN (BTW I was a high school leaver) I would expect any CS graduate to be able to do that - either that or I am a 10x programmer and didn't realize it.
    • by cwebster ( 100824 ) on Saturday September 21, 2013 @05:00PM (#44913783)

      Please don't learn FORTRAN, learn Fortran instead. (For the pedantic, all caps is F77. Normal caps is F90 and later.)

  • by poodlediagram ( 1944244 ) on Saturday September 21, 2013 @01:48PM (#44912825)

    ...and has done for years.

    We write a scientific code for solving quantum mechanics for solids and use both OpenMP and MPI in hybrid. Typically we run it on a few hundred processors across a cluster. A colleague extended our code to run on 260 000 cores sustaining 1.2 petaflops and won a supercomputer prize for this. All in Fortran -- and this is not unusual.

    Fortran gets a lot of bad press, but when you have a set of highly complex equations that you have to codify, it's a good friend. The main reason is that (when well written) it's very easy to read. It also has lot's of libraries, it's damn fast, the numerics are great and the parallelism is all worked out. The bad press is largely due to the earlier versions of Fortran (66 and 77), which were limited and clunky.

    In short, the MPI parallelism in Fortran90 is mature and used extensively for scientific codes.

    • If your code is already parallelized, LibGeoDecomp might not have a terrible lot to offer for you. The blog post was by no means directed against Fortran as a language. Instead it advocates a way for folks to bring their existing, sequential Fortran codes to supercomputers without having to spend months doing the parallelization manually.
  • by msobkow ( 48369 ) on Saturday September 21, 2013 @02:26PM (#44912991) Homepage Journal

    You don't have to rewrite your code entirely, just a little bit.

    You only have to restructure the subroutines and change the syntax.

    Well, that sounds like rewriting to me. Just because there is a library that might implement the same semantics as FORTRAN's math does not mean that it isn't a rewrite, coming with all the risks for new errors and gotchas that that implies.

    • Just asking because otherwise you'd had a better view on how intrusive (or not) this restructuring is. To give some numbers: a while ago we ported a simulation (video here [youtube.com]) to the library. The simulation model was about 5000 lines of code. Not much, but the code was highly condensed and had been carefully modeled in the course of 3 years. We ended up having to change less than 100 lines to make it work with LibGeoDecomp. That's a far cry from a rewrite.
      • by msobkow ( 48369 )

        And so on the basis of one example you're willing to take their word that changing languages doesn't require re-debugging the entire program?

        My, my, but you are naive, aren't you?

        • by msobkow ( 48369 )

          Even when you change compilers but keep the same source code you have to redebug complex FORTRAN code, due to idiosyncracies in implementations over the years.

        • by msobkow ( 48369 )

          Sometimes you get new bugs even with the same compiler, just because you changed optimization flags for the build.

        • Sorry, I should probably have added a disclaimer that I'm involved in the development of the library as my signature apparently doesn't make it obvious enough: I'm the project lead.

          So far we've built about a dozen application with LibGeoDecomp, including porting a dozen large scientific codes towards it. You're right that porting a code usually involves debugging. But that's inevitable when parallelizing a previously sequential code anyway. We don't claim to do magic, we just have some cool tricks up our s

  • Someone has some legacy Fortran code and a task of modifying it. There are two approaches: Port it or work on the existing source. Porting it allows for hiring from a very large (but shallow*) pool of programmers familiar with 'current' languages like C++. Working with the existing code means having to locate resources in a much smaller market. The former are cheap. The latter much more expensive. What to do?

    *Good programmers can probably pick up a book and teach themselves Fortran pretty easily. But eve

  • Reminds me of a recent experience writing a new system to replace a legacy system.

    A key part of one of the homegrown network protocols was a CRC. This sounds OK, but the implementation was wrong. I spent a lot of time trying to reverse-engineer just what the original engineers had implemented. The fact that it was written in ADSP2181 assembler didn't help. It had never been an issue before because both ends of the link used the same wrong implementation, so the errors cancelled out.

    I ended up writing an

  • by excelsior_gr ( 969383 ) on Saturday September 21, 2013 @03:02PM (#44913137)

    It is true that there are a lot of legacy Fortran codes in scientific computing, but chances are that they are already parallel, so this tool won't be much of a use for those supporting them. OpenMP and MPI have been in use in Fortran codes for decades. The summary seems to think that legacy Fortran codes need saving and porting. They don't. They are just fine, number crunching faster than you can say DO CONCURRENT.

    Having said that, LibGeoDecomp seems quite nice if you find a piece of serial code and you want to make a rough parallel version of it without much hassle. But if you are writing new code, you can parallelize it natively. Nevertheless, I believe that we must focus our resources in developing the current compilers. The Compaq compiler died in the hands of HP and people moved mostly to the intel compiler, since the open-source community was focused in C++ at the time and the gcc was stuck with the obsolete g77. Then g95 came along, that brought us all the cool stuff of Fortran 90/95, while gfortran was being developed. Now gfortran seems decent, but it still has to match the speed of ifort in order to sit at the cool kids' table. Also, we need the features of the latest Fortran standards. I would gladly use a compiler that is feature-complete, even if the executables are relatively slow, because I will be able to switch into the mindset of the Fortran2008 standard and stop doing things the Fortran95-way while coding. They will then have all the time they need to make it more efficient.

  • by tlambert ( 566799 ) on Saturday September 21, 2013 @03:10PM (#44913193)

    In my personal experience...

    Most of the physics code in FORTRAN that I've dealt with are things like relativistically invariant P-P and N-P particle collision simulations in order to test models based on the simultaneous solution to 12 or more Feynman-Dyson diagrams. It's what was used to predict the energy range for the W particle, and again for the Higgs Boson, and do it rather reliably.

    The most important part of this code was reproducibility of results, so even though we were running Monte Carlo simulations of collisions, and then post-constraining the resulting pair productions by the angles and momentum division between the resulting particles, the random number stream had to be reproducible. So the major constraint here was that for a reproducible random stream of numbers, you had to start with the same algorithm and seed, and the number generation had to occur linearly - i.e. it was impossible to functionally decompose the random number stream to multiple nodes, unless you generated and stored a random number stream sufficient to generate the necessary number of conforming events to get a statistically valid sample size.

    So, it was linear there, and it was linear in several of the sets of matrix math as it was run through the diagrams to filter out pair non-conforming pair production events.

    So we had about 7 linearity choke-points, one of which could probably be worked around by pre-generating a massive number of PRNG output far in excess of what would be eventually needed, and 6 of which could not.

    The "add a bunch of PCs together and call it a supercomputer" approach to HPC only works on highly parallelizable problems, and given that we've had that particular capability for decades, the most interesting unsolved problems these days are not subject to parallel decomposition (at least not without some corresponding breakthroughs in mathematics).

    I converted a crap-load of FORTRAN code to C in order to be able to optimize it for Weitek vector processors plugged into Sun hardware, including the entire Berkeley Physics package, since that got us a better vector processor than was on the Cray and CDC hardware at Los Alamos where the code was running previously, but adding a bunch of machines together would not have improved the calculation times.

    Frankly, it seems to me that the available HPC hardware being inherently massively parallel has had a profound effect on constraining the problems we try to solve, and that there are huge, unexplored areas that are unexplored for what amounts to the equivalent of someone looking for their contact lens under the streetlight, rather than in the alley where they lost it, "because the light's better".

    • You're right: the current compute architectures we see in HPC are geared at data parallel problems of massive size. Clock speeds are stagnating, sometimes even stepping down (e.g. NVIDIA Kepler has its cores actually clocked slower that Fermi with its hot clock for the shaders). Your description sounds like you'd benefit from a singular core which is tuned for single thread performance (e.g. with really big caches, a large out of order execution window) and runs at 5-10 GHz (which might require liquid nitro

  • Source code modification is required, but mostly limited to restructuring into a new pattern of subroutines.

    That's not what I call "limited". More like a rewrite, or at least a salvage operation.

After all is said and done, a hell of a lot more is said than done.

Working...