Microsoft Roslyn: Reinventing the Compiler As We Know It 195
snydeq writes "Fatal Exception's Neil McAllister sees Microsoft's Project Roslyn potentially reinventing how we view compilers and compiled languages. 'Roslyn is a complete reengineering of Microsoft's .NET compiler toolchain in a new way, such that each phase of the code compilation process is exposed as a service that can be consumed by other applications,' McAllister writes. 'The most obvious advantage of this kind of "deconstructed" compiler is that it allows the entire compile-execute process to be invoked from within .NET applications. With the Roslyn technology, C# may still be a compiled language, but it effectively gains all the flexibility and expressiveness that dynamic languages such as Python and Ruby have to offer.'"
3 years ago (Score:2)
What do they exactly mean by "flexibility and expressiveness of other dynamic languages" ?
I remember a demo at a Microsoft Developer congress where C# would be able to execute and rebuilt itself dynamically.
At the time it got me really excited (as I've bumped into many problem which would have a much more beautiful solution should I be able to compile during runtime.) but this seems yet another technology?
Re: (Score:3)
Re: (Score:2)
I'm wondering if my programming skills have fallen away so much through lack of use that I don't understand this as well anymore, or if the summary/article is just full of buzzwords and impressive sounding jargon.
It's entirely the latter. What they're describing is nothing Lisps haven't been doing for decades. Although I don't need to use in my everyday programming, standard Python also exposes APIs for compiling source and manipulating syntax trees.
Re: (Score:2)
Lisp was there first in many cases, yet to the majority of the programming world, it's just a bunch silly parenthesis (nevermind that semicolons and curly braces everywhere are just as bad if not worse).
I'm fully aware that most programmers are unaware of the significance of Lisp, which is why I also gave mentioned Python, which is far more mainstream. Lisps were only the first to implement the features described in TFA; they've been implemented in many other languages since then.
Re: (Score:2)
Re: (Score:3)
What do they exactly mean by "flexibility and expressiveness of other dynamic languages" ?
I remember a demo at a Microsoft Developer congress where C# would be able to execute and rebuilt itself dynamically.
At the time it got me really excited (as I've bumped into many problem which would have a much more beautiful solution should I be able to compile during runtime.) but this seems yet another technology?
I don't think of it as a new technology, but Microsoft is finally getting around to it. They are such a big dog now that some people don't recognize change until Microsoft rolls it out - years after others have already been mucking about in it for years.
If they roll it out in a good package, that's a good thing. If they price it above most developers budgets than they're going to be bypassed.
Re: (Score:2)
They'll probably do exactly what they've done with the past 2 or 3 releases of Visual Studio.
Roll it out, have something that's actually pretty damn good, and as long as you aren't making a commercial product, it is all free (except for a couple reporting tools and their code repository server, the latter of which can easily be replaced with 90% of the features by using Mercurial using VisualHG + TortoiseHG, which are free and better suited for most non-corporate development models anyway).
Now, if you want
Re: (Score:2)
I'm not about to debate anyone on the Visual Studio segmented pricing, but Anders Hejlsberg has gone on record saying he'd like for Roslyn to be open source, by which he likely means Ms-PL. If they do that, they could indeed close it later, at the cost of an enormous shitstorm. (When Microsoft has stopped providing open source before, they've stopped developing the software (see: Rotor). This is the new version of the C# compiler and its IDE integration so that's not going to happen.)
Re: (Score:2)
I thought there was a limit of one or two commercial projects from VS Express.
Re: (Score:2)
It is no surprise whatsoever. TFS is very much modelled after P4, but P4 is considerably more mature and robust - and it's not even terribly expensive either.
Re: (Score:2)
Different groups use different tools, many new projects use TFS features, older ones use an internal tool, I havn't heard of anyone using Perforce, but I havn't been looking particularly.
Re: (Score:2)
years after others have already been mucking about in it for years
Could you provide an example of that? This is not the ability to compile code at runtime, .NET has had that for years. I have several .NET projects where I generate code on the fly, compile and then run it. Have the same for Java. That is not what Roslyn is.
Re: (Score:2)
Very few would need more features than the professional version, which is $750 or so a seat from many retailers. Still steep for a lot of users, but unless you are a huge software corporation, Ultimate is overkill.
Re: (Score:2)
Re: (Score:2)
Gee, it looks like $3,799.00 to me, must be the font I use
Or maybe you have an MSDN subscription already? That's the price for renewal, the grandparent's price is the one for a new subscription.
Re: (Score:2)
That price includes an MSDN subscription, no surprise that you're an AC.
Re: (Score:2)
This is what I used (and later programmed myself) about 20 years ago in LPC (Lars Pensjö C, later called Pike).
It was a command called "lpc", and it just took a piece of code as argument, wrote it into a file, compiled it on the fly and called its main and only method.
Re: (Score:2)
In the past, you could compile C# source code at runtime and load it into memory and use it. (I know, I was doing it for a MUD that didn't pan out.) But it wasn't very dynamic. (We were using it to reload the code without restarting the MUD.)
I assume this news means that you can write code dynamically, without writing it to a text file first.
Re: (Score:2)
Ah, we didn't try that because we didn't need it. I wasn't 100% sure if it was possible or not. (Plus, I don't remember much about it.)
Anyhow, since they were so specific about each piece being separate, I'm thinking that things might be a lot more flexible this new way... Maybe not even writing code at all, but just creating it programatically. We'll see.
Re:3 years ago (Score:5, Insightful)
Tiny C compiler does this for years:
http://bellard.org/tcc/ [bellard.org]
Features
SMALL! You can compile and execute C code everywhere, for example on rescue disks (about 100KB for x86 TCC executable, including C preprocessor, C compiler, assembler and linker).
FAST! tcc generates x86 code. No byte code overhead. Compile, assemble and link several times faster than GCC.
UNLIMITED! Any C dynamic library can be used directly. TCC is heading torward full ISOC99 compliance. TCC can of course compile itself.
SAFE! tcc includes an optional memory and bound checker. Bound checked code can be mixed freely with standard code.
Compile and execute C source directly. No linking or assembly necessary. Full C preprocessor and GNU-like assembler included.
C script supported : just add '#!/usr/local/bin/tcc -run' at the first line of your C source, and execute it directly from the command line.
With libtcc, you can use TCC as a backend for dynamic code generation.
Re: (Score:2, Interesting)
You might want to read this wrt tinycc: http://www.landley.net/code/tinycc/
Also, no x86-64.
Re: (Score:2)
You might want to read this wrt tinycc: http://www.landley.net/code/tinycc/ [landley.net]
Jesus. That is the saddest story I've ever read. I want to find that guy and give him a hug.
That's a pretty good summary of why I never bother contributing to open source any more, even though I have tons of code lying around including cool modifications of some pretty big-name projects. I've been lucky enough to never work with douchebags in the salaried, closed source world. I sure as hell will not suffer it while also not makin
Re: (Score:2)
wow that sucks balls.
If I were him, I would have just given up on them much sooner. I used CVS myself exclusively until recently but, I don't think I ever had the illusions of it being really good or even sufficient for real development (luckily for most of my use cases, cvs is borderline overkill)
I can't imagine someone using CVS heavily, for any significant period of time, and still believeing it to be the bees knees. Unless you have a top notch CVS admin who swoops in fixes things, and makes the annoying
Re:3 years ago (Score:5, Informative)
Re: (Score:3)
The code is here [slashdot.org]. The AST / back end are in LanguageKit, the Smalltalk front end is in Smalltalk (this also contains a few support things that make OpenStep classes look a bit more like Smalltalk-80 ones). The JavaScript-like language is in EScript, but it may not be working at the moment. It currently requires a trunk build of GNUstep libobjc [gna.org], but I plan on releasing 1.6 of the runtime Real Soon Now.
I periodically write things about it on the Étoilé blog [etoileos.com]. You can also read some slightly ou [swan.ac.uk]
Re: (Score:2)
I think the idea is that it will be easier than the current methods. Also, if I remember correctly from previous articles, there will be more profiling information available to help with optimization tasks.
Re: (Score:2)
yes, but a lot of that was built through the reflection feature of .NET, which in turn makes it very slow and clumsy. Reflection has its uses, but the easiest way to tell is to use the following rule: "if you're using it you're doing things wrong."
Re: (Score:2)
old news? (Score:4, Interesting)
Re: (Score:2)
That was the announcement of the release. This was the actual release.
Frigging newbies..... (Score:2)
http://www.occupytheboardroom.org/ [occupytheboardroom.org]
The compiler as I know it (Score:2, Insightful)
If I wanted to, I could rig GCC and the like to do that too: That's the wonderful thing about command-line tools and piping, you can munge things together any way you want. And of course you can always tell gcc to stop partway through the compilation if you need assembler code or a parse tree or something. This sort of thing is common in open-source compilers, because they need these features for debugging purposes and have no reason to leave them out of the released version.
Of course, I probably don't want
Re: (Score:2)
Yeah, just try it wtih GCC. Not easy. CLANG on the other hand is looking AWESOME for this sort of thing. Once a libraryized version of CLANG is good, it'll make this sort of thing easy.
Re: (Score:3)
Uh, clang is 'libraryized'. The clang binary is a tiny wrapper around the various libraries. It's pretty simple to write a replacement or to embed the libraries in something else. Look at Cling, for example, which implements a C++ REPL system using the libraries, or LLDB, which uses clang to parse [Objective-]C[++] expressions in the debugger.
If the grandparent thinks any of this is easy with gcc, then he's never tried hacking on gcc - even using it for syntax highlighting is almost impossible because t
Re: (Score:2)
The "dynamic code execution" just builds on top of support that's already there. You could build a string of a C# program, feed it to a compiler and run it dynamically previously as well. This just makes the compilation part less of a headache.
Rigging GCC (other comments have already brought up clang) would maybe be more dangerous because arbitrary C code could do anything. Arbitrary IL code can be hamstrung by some security layers. Of course, the differences between managed and native code pale in comparis
Re: (Score:2)
This is about much more than just a way to run the compiler. It exposes all lexer and parser layers, so you can get AST. It also exposes separate stages of semantic analysis of that tree, so you can e.g. get it with type annotations for all nodes (after all type inference and method overload resolution), or metadata specifying which of the three foo's in scope this particular identifier "foo" refers to.
Basically, it's a toolkit that lets you write software that works with C# or VB code and requires deep und
Never heard of Clang? (Score:5, Informative)
Re: (Score:2)
Apple is making use of LLVM and Clang in their IDE for exactly the kinds of things talked about in the article, replacing custom parsers used for syntax highlighting or expression parsing in the debugger.
Nokia recently adopted Clang too for Qt Creator:
http://labs.qt.nokia.com/2011/10/19/qt-creator-and-clang/ [nokia.com]
Re: (Score:2)
I wonder if older compilers like GCC will be able to adapt or if they'll just continue being monolithic.
GCC intentionally resisted clean separation and layering because someone might do something evil like create a GPL'd program that did syntax highlighting and invoke it via a pipe from a proprietary program. As a side effect, this made it impossible to integrate into Free IDEs[1]. In response to LLVM and Clang being vastly better for this kind of thing, the GCC team has finally allowed you to write plugins, but it's a bit late. About the only reason to use GCC these days is if you really need to target on
gift culture Lebensraum (Score:2)
I've been around a long time, and I've never heard that. It has the kind of plausible ring that usually sends me to Snopes, where two thirds of the time I come away chastised for loaning the idea five seconds of credence.
What I know about GCC is that it had a rough adolescence and that over-arching desi
Re: (Score:2)
Futhermore, on the C++ side, the rapid evolution of the C++ language wasn't doing anyone any favours in IDE integration
Shouldn't the word rapid be in quotes here?
Interactive Compilation (Score:2)
Re: (Score:2)
Boo (boo.codehaus.org) lets you work this way... It's a Python-esque, type-safe scripting language for .Net...
Great for a tiny minority, meh for everyone else (Score:2)
This sounds great if you're doing stuff like autotuning, but for the vast (vast, vast, vast) majority of programmers out there I don't really see how opening up the internals of the compiler is useful. Who cares if that loop gets fused or that function gets unrolled?
Sounds like LLVM (Score:3, Informative)
Roslyn is a complete reengineering of Microsoft's .NET compiler toolchain in a new way, such that each phase of the code compilation process is exposed as a service that can be consumed by other applications,
Sounds like LLVM.
Re: (Score:2)
Sounds like LLVM.
Eric Lippert's responce to that suggestion is "absolutely not". http://blogs.msdn.com/b/ericlippert/archive/2011/10/19/the-roslyn-preview-is-now-available.aspx [msdn.com]
opengtl, llvm, krita (Score:2, Informative)
LISP had that 40 years ago (Score:5, Informative)
This isn't exactly new. LISP had it from the early days. It's an idea that's been tried before, now available with more modern buzzwords, like "the compiler as a service".
Re: (Score:3)
You can develop for Metro in .Net after the last announcement, and it will be supported on ARM.
Re: (Score:2)
Re: (Score:3)
Lisp does not have any syntax to speak of, so you don't need a large framework to do syntactic analysis. And it's dynamically typed, so most semantic analysis is rather pointless.
What this does is give you a annotated (with types, resolved references etc) code model that you can build upon. It's not at all like, say, CL macros. A different tool for a different job - think static code analysis, advanced refactoring, code-aware search etc.
Re: (Score:2)
The difference is that LISP is an interpreted language.
Re: (Score:2)
No matter how many times it's been tried before, it's still a neat idea. It'll be interesting to see where Microsoft takes this idea.
Re: (Score:2)
True AI requires the ability to rewrite the code of the AI system itself, thereby implementing learning algorithms. My own work is primitive compared to a an AI; I've only created an expert system, which is a much simpler thing.
A true programming AI would be able to extract data structures and algorithms from virtually any language, learn from it, and teach itself to program in those languages, rather than having someone manually teach it by updating the knowledge base.
Re: (Score:2)
the java API allows you to access/modify the syntax tree, get data-flow analysis, etc...? or is it just a dumb wrapper around the compiler front-end?
Sounds cool (Score:2)
Can't wait to get my hands on a FOSS clone of it.
Scala (Score:2)
like the Scala compiler? an API, plugin support and more? the Scala shell uses it as an example of how to use it
Compiled vs. Dynamic? (Score:2)
C#, Ruby, and Python are all (in their main implementations) compiled languages. Where they differ is that C# is mostly-statically-typed, and Ruby and Python are dynamically-typed. The .NET compiler toolchain being exposed as a runtime service doesn't really make C# much more like Ruby or Python, since it doesn't change
Re: (Score:2)
C# got optional dynamic typing in version 4 with the DLR, although the DLR is really a set of libraries that just helps with building the required expression tree at run time to allow the dynamic behaviour.
I do somewhat agree though that I don't see how this gives it any more of the benefits of Python and Ruby than it already had.
Re: (Score:2)
Right. That's why I said it was mostly-statically-typed rather than just plain statically-typed.
Compiler-as-a-service is a nice feature for the .NET runtime (and, as I understand it, this is a .NET runtime feature, not a C# language feature), but the Ruby/Python comparison doesn't really seem to be on-point as to why.
Re: (Score:2)
Ruby's primary implementation is a bytecode compiler and a runtime VM..
That has nothing to do with compiled vs. interpreted, but with the semantics of method call resolution.
EUREKA! (Score:2)
They found a way to shove XML into the compiler! Kudos to MS!
(see sig)
It's not a new idea (Score:2)
For example, code indentation can be done by walking the AST (you need to be careful to preserve hidden tokens, such as comments).
You can also allow code completion by changing the compiler to accept a "COMPLETION" token in some places in the grammar. Then, from the editor, when someone presses "Ctrl+SPACE" (or whatever) you mark the location in the lexer
Managed AST + Managed Compiler == Boo (Score:2)
This premise, a managed AST you can manipulate programmatically (a SOM, Source Object Model), plus a managed compiler pipeline to compile, is nothing new. Boo language was doing this on .NET , and I'm sure there are many examples before it: Boo was started in 2003.
Gasping for air (Score:2)
Seems to me Microsoft is now attempting to do with compilers what they attempted to do with the mobile phone.
Javassist? (Score:2)
How is this any different at all from Javassist?
Java Already Does This (Score:2)
I see a lot of other tools that do this, but since C# mostly started off as a ripped off Java, it's also worth pointing out that since Java 1.6, that language also provided public interfaces to compile code at runtime [oracle.com].
These are nice features. Sometimes, they are even useful (as opposed to just another hammer developers can abuse). But the announcement makes it seem, wrongly, that MSFT is doing something really unique here.
Re: (Score:2)
LOL!
C# Has had the equivalent of javax.tools.JavaCompiler since v1.1, Microsoft.CSharp.Compiler [microsoft.com] which is little more than a wrapper around the command-line compiler.
But the .NET framework has also, for a long time, included Reflection.Emit [microsoft.com] which allows for direct manipulation of CIL bytecode, and System.CodeDom [microsoft.com] for generating and compiling source code in multiple languages from an abstract representation.
I wouldn't bother reading about Roslyn, though. It couldn't possibly measure up to anything Java has had
Nothing new here for c# (Score:2)
C# has always supported compiling additional code at runtime.
I've had it in projects since the 1.0 release.
They may be redoing the structure and making it easier to do, but doing it isn't new.
Re: (Score:2)
doesn't this allow for malicious programs to get even more malicious?
If some weasel can figure a way to insert malicious code between Source and Executable, there's always the possibility (and always has been.)
Re: (Score:2)
Isn't the point of having the compiler as a service so that your executable can feed source code to a compiler?
Re: (Score:3)
Filling in a few blanks and tweaks, that can be done on ANY unix system. It allows ANY software to feed source code to a compiler. Nobody has complained of this as a security risk before.
Now, it might be a *slight* security risk if it is running as a background process that is always on, and therefore corrupting it once could potentially corrupt all future output, but I
Re:security? What about.... (Score:2)
How bout if the 'other app' is a web browser window? TFA suggests this will be possible with MS's product.
Re: (Score:3)
So? my code could be put in an apache module. Use WSGI and it is available in Python. PHP has the ability to do it straight away.
It's still not adding any vulnerabilities to the ecosystem that haven't existed before. Yes they used it as a demo, but that's probably because it's a quickly visible demo that everyone can easily see what it is doing. Only an idiot would use it like that on a production system, just like only an idiot would use C, PHP or Python to do the same thing, and those have had that featur
Re: (Score:2)
Only an idiot would use it like that on a production system
Would you call a JIT compiler idiotic then ? Because this is exactly how I foresee this stuff being used in enterprise apps, particularly ones that rely heavily on dynamic entities. We could have the app generate code on-the-fly that is then reused as needed, rather than reinterpreted every time with a hundred DB calls and long-winded generic form-generating code.
Re: (Score:2)
Please read the previous comments, a JIT compiler is NOT what we are discussing. Those are used to compile static, pre-existing code.
Re: (Score:2)
Addendum, didn't read your whole comment, still not what we were discussing. We were discussing the use of dynamic code entered by the user making a request via the web, and compiled/executed by the application. Not a specific set of pre-defined templated code that is modified without taking code directly or indirectly from the user.
Re: (Score:2)
There are already APIs to emit IL or to invoke a C# compiler built into .NET and the security systems built into .NET give you a way to prohibit them. There's no additional risk exposed by Roslyn. Rather, it's a way of getting at the juicy knowledge about the code that the compiler builds up before it exits and that libraries have been written to poorly piece together. That's a good idea that I'd like to see accompany more official language compilers, static or not.
Re: (Score:3)
This is the reason why gcc (or any other compiler) should never be installed on any production Linux machine.
Having a compiler installed permits the add-hoc creation of code, with all the resultant security risks. Self-modifying code, including compiled self-modifying code, is an elegant solution in certain environments, however it is a huge security and reliability risk in any production application.
The problem with the Windows NT 4.0 security model was that security was present almost everywhere, excep
Re: (Score:2)
By that logic, Python, PHP, Perl and any other scripting language that can popen() or eval() should not be a
Re: (Score:2)
Yeah, you could write a C++ compiler in PHP and use that instead of gcc.
Re: (Score:2)
I'm simply talking about popen() and exec() like features. Loading of dynamic libraries, etc.
You can have pretty much the same effect without having to resort to native compiled code.
Re: (Score:2)
Bogus response. Compilers are on programmer's computers and they are used in production and suffer the same potential as you describe.
Re: (Score:2)
Having a compiler installed permits the add-hoc creation of code
wait, if gcc is a security risk, then so is chmod + (any program capable of writing binary data to a file). the only 'vulnerability' that gcc exposes is the ability to create cross-architecture exploits.
Re: (Score:2)
But this feature will be encouraging developers to write applications that accept source at run time. If you don't write that into your app, you only have to protect it from malicious code before compilation, in most cases. Which is much, much easier.
Of course there are always other security threats after compilation as well, and those will still be there in addition to the ones this opens up.
Re:security? My first thot, too... (Score:2)
Re: (Score:3)
platform-independence for malware just got easier!
Re: (Score:2)
Re: (Score:2)
Oh?
I run it on Windows, Linux, FreeBSD and MacOS.
I run it on x86 and ARM.
Seems pretty damn independent to me.
Regarding what the GP stated though, with the right libraries and a little clever coding, a similar independent 'partially compiled' method could be used with C as well. Of course the partial compile of the windows version would have to check for a C compiler, and download/install one if it isn't available. Java and Flash could conceivably be used to do the same. So, it's really not adding a whole l
Re: (Score:2)
It allows for something similar to eval [wikipedia.org] in .NET. From the article:
Re: (Score:2)
Consider cases like the console interpreter for Python or Perl, you could do malicious things with them, but I wouldn't call them code injection attacks. Mind you, using a REPL in a more complex piece of code that has other functions is probably a bad idea, but I think the REPL is more of an easy demo to show what it can do, not the intended use.
Re: (Score:2)
Yes, that's true. When I talked about injection attacks I was thinking more about using this to run JSON-like strings of code when you don't trust the source.
Re: (Score:2)
Re: (Score:2)
They may not necessarily mean 'service' as runtime service (the Windows world equivalent of a Daemon), but rather more a sense of a software library (provides a service to other applications, without actually being standalone itself).
In this case, it's not much different from python/ruby on any system. I could execute arbitrary python code from a C executable without much difficulty. That doesn't mean that there is a problem with how, C, Python or the underlying OS is written.
Re: (Score:2)
It is not possible to reinvent. That's an oxymoron. They are simply building upon what they and others have developed. Radical technology change, yes, but you can't reinvent anything, especially a compiler.
Re: (Score:2)
Really? I like the REPL, but I wasn't aware that they had fixed the entanglement issues.
Thanks to Roslyn being designed explicitly for these kinds of scenarios, it can give you helpful information from nearly every stage of the compilation progress. You can get syntax trees! Not only that, you can feed it an invalid program and you'll get back a syntax tree that says that it's invalid, but knows when it stopped parsing, what kind of token it expected and can be stringified to the exact text you fed it. You
Re: (Score:2)
The Csharp REPL is really just an example usage of Mono.CSharp to which I should have linked to instead.
Re: (Score:2)
Sure, but I meant the compiler when I said that, which I thought was clear from the third paragraph.
The Mono C# compiler can't do all this stuff, and that's completely fine because it's almost impossible to do by accident. I don't fault them for not having done it by accident. If anything, they should be commended for being able to whip up the REPL so easily; that shows commitment to solid design principles.
But that also means that just staying where they are thinking that they have parity, or someone propo
Re: (Score:2)
Re: (Score:2)
The .NET framework includes the C# compiler for free, csc.exe.
The ability to compile C# code in .NET apps has been available since before 1.0 release. This isn't new, just a different way of doing it.