Follow Slashdot blog updates by subscribing to our blog RSS feed

 



Forgot your password?
typodupeerror
Programming Microsoft

Microsoft Roslyn: Reinventing the Compiler As We Know It 195

Posted by Soulskill
from the now-with-3d dept.
snydeq writes "Fatal Exception's Neil McAllister sees Microsoft's Project Roslyn potentially reinventing how we view compilers and compiled languages. 'Roslyn is a complete reengineering of Microsoft's .NET compiler toolchain in a new way, such that each phase of the code compilation process is exposed as a service that can be consumed by other applications,' McAllister writes. 'The most obvious advantage of this kind of "deconstructed" compiler is that it allows the entire compile-execute process to be invoked from within .NET applications. With the Roslyn technology, C# may still be a compiled language, but it effectively gains all the flexibility and expressiveness that dynamic languages such as Python and Ruby have to offer.'"
This discussion has been archived. No new comments can be posted.

Microsoft Roslyn: Reinventing the Compiler As We Know It

Comments Filter:
  • What do they exactly mean by "flexibility and expressiveness of other dynamic languages" ?

    I remember a demo at a Microsoft Developer congress where C# would be able to execute and rebuilt itself dynamically.

    At the time it got me really excited (as I've bumped into many problem which would have a much more beautiful solution should I be able to compile during runtime.) but this seems yet another technology?

    • I'm wondering if my programming skills have fallen away so much through lack of use that I don't understand this as well anymore, or if the summary/article is just full of buzzwords and impressive sounding jargon.
      • by Jonner (189691)

        I'm wondering if my programming skills have fallen away so much through lack of use that I don't understand this as well anymore, or if the summary/article is just full of buzzwords and impressive sounding jargon.

        It's entirely the latter. What they're describing is nothing Lisps haven't been doing for decades. Although I don't need to use in my everyday programming, standard Python also exposes APIs for compiling source and manipulating syntax trees.

    • by ackthpt (218170)

      What do they exactly mean by "flexibility and expressiveness of other dynamic languages" ?

      I remember a demo at a Microsoft Developer congress where C# would be able to execute and rebuilt itself dynamically.

      At the time it got me really excited (as I've bumped into many problem which would have a much more beautiful solution should I be able to compile during runtime.) but this seems yet another technology?

      I don't think of it as a new technology, but Microsoft is finally getting around to it. They are such a big dog now that some people don't recognize change until Microsoft rolls it out - years after others have already been mucking about in it for years.

      If they roll it out in a good package, that's a good thing. If they price it above most developers budgets than they're going to be bypassed.

      • by ByOhTek (1181381)

        They'll probably do exactly what they've done with the past 2 or 3 releases of Visual Studio.

        Roll it out, have something that's actually pretty damn good, and as long as you aren't making a commercial product, it is all free (except for a couple reporting tools and their code repository server, the latter of which can easily be replaced with 90% of the features by using Mercurial using VisualHG + TortoiseHG, which are free and better suited for most non-corporate development models anyway).

        Now, if you want

        • by wootest (694923)

          I'm not about to debate anyone on the Visual Studio segmented pricing, but Anders Hejlsberg has gone on record saying he'd like for Roslyn to be open source, by which he likely means Ms-PL. If they do that, they could indeed close it later, at the cost of an enormous shitstorm. (When Microsoft has stopped providing open source before, they've stopped developing the software (see: Rotor). This is the new version of the C# compiler and its IDE integration so that's not going to happen.)

      • by terjeber (856226)

        years after others have already been mucking about in it for years

        Could you provide an example of that? This is not the ability to compile code at runtime, .NET has had that for years. I have several .NET projects where I generate code on the fly, compile and then run it. Have the same for Java. That is not what Roslyn is.

    • by Aladrin (926209)

      In the past, you could compile C# source code at runtime and load it into memory and use it. (I know, I was doing it for a MUD that didn't pan out.) But it wasn't very dynamic. (We were using it to reload the code without restarting the MUD.)

      I assume this news means that you can write code dynamically, without writing it to a text file first.

    • Re:3 years ago (Score:5, Insightful)

      by sourcerror (1718066) on Friday October 21, 2011 @11:01AM (#37793662)

      Tiny C compiler does this for years:
      http://bellard.org/tcc/ [bellard.org]

      Features

      SMALL! You can compile and execute C code everywhere, for example on rescue disks (about 100KB for x86 TCC executable, including C preprocessor, C compiler, assembler and linker).

      FAST! tcc generates x86 code. No byte code overhead. Compile, assemble and link several times faster than GCC.

      UNLIMITED! Any C dynamic library can be used directly. TCC is heading torward full ISOC99 compliance. TCC can of course compile itself.

      SAFE! tcc includes an optional memory and bound checker. Bound checked code can be mixed freely with standard code.
      Compile and execute C source directly. No linking or assembly necessary. Full C preprocessor and GNU-like assembler included.

      C script supported : just add '#!/usr/local/bin/tcc -run' at the first line of your C source, and execute it directly from the command line.

      With libtcc, you can use TCC as a backend for dynamic code generation.

      • Re: (Score:2, Interesting)

        by Anonymous Coward

        You might want to read this wrt tinycc: http://www.landley.net/code/tinycc/
        Also, no x86-64.

        • by pclminion (145572)

          You might want to read this wrt tinycc: http://www.landley.net/code/tinycc/ [landley.net]

          Jesus. That is the saddest story I've ever read. I want to find that guy and give him a hug.

          That's a pretty good summary of why I never bother contributing to open source any more, even though I have tons of code lying around including cool modifications of some pretty big-name projects. I've been lucky enough to never work with douchebags in the salaried, closed source world. I sure as hell will not suffer it while also not makin

          • by TheCarp (96830)

            wow that sucks balls.

            If I were him, I would have just given up on them much sooner. I used CVS myself exclusively until recently but, I don't think I ever had the illusions of it being really good or even sufficient for real development (luckily for most of my use cases, cvs is borderline overkill)

            I can't imagine someone using CVS heavily, for any significant period of time, and still believeing it to be the bees knees. Unless you have a top notch CVS admin who swoops in fixes things, and makes the annoying

      • Re:3 years ago (Score:5, Informative)

        by TheRaven64 (641858) on Friday October 21, 2011 @01:24PM (#37796182) Journal
        LLVM is a better example. It's a set of libraries for generating a code in an intermediate representation, transforming that representation (usually for optimisation, but also for instrumentation and other things) and then emitting it as object code, assembly, or JIT'd executable code in memory. I've written compilers for Smalltalk and for a toy JavaScript-like language using it, and they share the same set of optimisations that I wrote for Objective-C and the same object model. The total amount of Smalltalk-specific code is about 15KLoC (including comments).
    • by ByOhTek (1181381)

      I think the idea is that it will be easier than the current methods. Also, if I remember correctly from previous articles, there will be more profiling information available to help with optimization tasks.

    • by gbjbaanb (229885)

      yes, but a lot of that was built through the reflection feature of .NET, which in turn makes it very slow and clumsy. Reflection has its uses, but the easiest way to tell is to use the following rule: "if you're using it you're doing things wrong."

    • by tixxit (1107127)
      It makes little sense, in that the dynamism of dynamic languages have nothing to do with "eval" (how many people write Python programs that write Python programs?). More over, runtime compilation is hardly new. Clojure, for example, is always compiled to bytecode, even at runtime, and is a dynamic language. Scala's REPL is a great example of a statically typed language being dynamically compiled and run. With Scala, you can even easily hook into the compiler tools to create your own interpreter and compile
  • old news? (Score:4, Interesting)

    by Verunks (1000826) on Friday October 21, 2011 @10:44AM (#37793360)
  • If I wanted to, I could rig GCC and the like to do that too: That's the wonderful thing about command-line tools and piping, you can munge things together any way you want. And of course you can always tell gcc to stop partway through the compilation if you need assembler code or a parse tree or something. This sort of thing is common in open-source compilers, because they need these features for debugging purposes and have no reason to leave them out of the released version.

    Of course, I probably don't want

    • Yeah, just try it wtih GCC. Not easy. CLANG on the other hand is looking AWESOME for this sort of thing. Once a libraryized version of CLANG is good, it'll make this sort of thing easy.

      • Uh, clang is 'libraryized'. The clang binary is a tiny wrapper around the various libraries. It's pretty simple to write a replacement or to embed the libraries in something else. Look at Cling, for example, which implements a C++ REPL system using the libraries, or LLDB, which uses clang to parse [Objective-]C[++] expressions in the debugger.

        If the grandparent thinks any of this is easy with gcc, then he's never tried hacking on gcc - even using it for syntax highlighting is almost impossible because t

    • by wootest (694923)

      The "dynamic code execution" just builds on top of support that's already there. You could build a string of a C# program, feed it to a compiler and run it dynamically previously as well. This just makes the compilation part less of a headache.

      Rigging GCC (other comments have already brought up clang) would maybe be more dangerous because arbitrary C code could do anything. Arbitrary IL code can be hamstrung by some security layers. Of course, the differences between managed and native code pale in comparis

    • This is about much more than just a way to run the compiler. It exposes all lexer and parser layers, so you can get AST. It also exposes separate stages of semantic analysis of that tree, so you can e.g. get it with type annotations for all nodes (after all type inference and method overload resolution), or metadata specifying which of the three foo's in scope this particular identifier "foo" refers to.

      Basically, it's a toolkit that lets you write software that works with C# or VB code and requires deep und

  • by gnasher719 (869701) on Friday October 21, 2011 @10:46AM (#37793400)
    It seems that Neil McAllister has never heard of LLVM and Clang, while Microsoft obviously has.
  • That's something that I haven't seen a language really get right since FORTH. I'd love to be able to use C# in a similar way, entering small function definitions from the command line, compiling them as they're entered, interactively testing functions as they're written. It's a great way to speed development.
    • by Forbman (794277)

      Boo (boo.codehaus.org) lets you work this way... It's a Python-esque, type-safe scripting language for .Net...

  • This sounds great if you're doing stuff like autotuning, but for the vast (vast, vast, vast) majority of programmers out there I don't really see how opening up the internals of the compiler is useful. Who cares if that loop gets fused or that function gets unrolled?

  • Sounds like LLVM (Score:3, Informative)

    by Dwonis (52652) * on Friday October 21, 2011 @10:56AM (#37793574)

    Roslyn is a complete reengineering of Microsoft's .NET compiler toolchain in a new way, such that each phase of the code compilation process is exposed as a service that can be consumed by other applications,

    Sounds like LLVM.

  • opengtl, llvm, krita (Score:2, Informative)

    by vurian (645456)
    Compile and execute code from within an application? That's exactly what Krita (http://www.krita.org) does with OpenGTL (http://opengtl.org) -- we have code written in special languages for filters and so on which gets compiled by Krita and then executed as native code. It's pretty safe as well.
  • by Animats (122034) on Friday October 21, 2011 @10:58AM (#37793606) Homepage

    This isn't exactly new. LISP had it from the early days. It's an idea that's been tried before, now available with more modern buzzwords, like "the compiler as a service".

    .NET as a virtual machine environment has become somewhat pointless, since the Windows/.NET environment is pretty much "write once, run on X86" now. Itanium support was dropped in 2010, and Microsoft's "Windows 8 for ARM" is apparently Metro-only.

    • You can develop for Metro in .Net after the last announcement, and it will be supported on ARM.

    • Lisp does not have any syntax to speak of, so you don't need a large framework to do syntactic analysis. And it's dynamically typed, so most semantic analysis is rather pointless.

      What this does is give you a annotated (with types, resolved references etc) code model that you can build upon. It's not at all like, say, CL macros. A different tool for a different job - think static code analysis, advanced refactoring, code-aware search etc.

    • by Hentes (2461350)

      The difference is that LISP is an interpreted language.

    • by msobkow (48369)

      No matter how many times it's been tried before, it's still a neat idea. It'll be interesting to see where Microsoft takes this idea.

      • by msobkow (48369)

        True AI requires the ability to rewrite the code of the AI system itself, thereby implementing learning algorithms. My own work is primitive compared to a an AI; I've only created an expert system, which is a much simpler thing.

        A true programming AI would be able to extract data structures and algorithms from virtually any language, learn from it, and teach itself to program in those languages, rather than having someone manually teach it by updating the knowledge base.

  • Can't wait to get my hands on a FOSS clone of it.

  • by robmv (855035)

    like the Scala compiler? an API, plugin support and more? the Scala shell uses it as an example of how to use it

  • With the Roslyn technology, C# may still be a compiled language, but it effectively gains all the flexibility and expressiveness that dynamic languages such as Python and Ruby have to offer.

    C#, Ruby, and Python are all (in their main implementations) compiled languages. Where they differ is that C# is mostly-statically-typed, and Ruby and Python are dynamically-typed. The .NET compiler toolchain being exposed as a runtime service doesn't really make C# much more like Ruby or Python, since it doesn't change

    • by Xest (935314)

      C# got optional dynamic typing in version 4 with the DLR, although the DLR is really a set of libraries that just helps with building the required expression tree at run time to allow the dynamic behaviour.

      I do somewhat agree though that I don't see how this gives it any more of the benefits of Python and Ruby than it already had.

      • C# got optional dynamic typing in version 4 with the DLR

        Right. That's why I said it was mostly-statically-typed rather than just plain statically-typed.

        Compiler-as-a-service is a nice feature for the .NET runtime (and, as I understand it, this is a .NET runtime feature, not a C# language feature), but the Ruby/Python comparison doesn't really seem to be on-point as to why.

  • They found a way to shove XML into the compiler! Kudos to MS!

    (see sig)

  • We used that approach for PBL [goo.gl] some years ago. It is wasteful to having to rewrite parsers and lexers for languages to build IDEs, and other tooling.

    For example, code indentation can be done by walking the AST (you need to be careful to preserve hidden tokens, such as comments).

    You can also allow code completion by changing the compiler to accept a "COMPLETION" token in some places in the grammar. Then, from the editor, when someone presses "Ctrl+SPACE" (or whatever) you mark the location in the lexer
  • This premise, a managed AST you can manipulate programmatically (a SOM, Source Object Model), plus a managed compiler pipeline to compile, is nothing new. Boo language was doing this on .NET , and I'm sure there are many examples before it: Boo was started in 2003.

  • Seems to me Microsoft is now attempting to do with compilers what they attempted to do with the mobile phone.

  • How is this any different at all from Javassist?

  • I see a lot of other tools that do this, but since C# mostly started off as a ripped off Java, it's also worth pointing out that since Java 1.6, that language also provided public interfaces to compile code at runtime [oracle.com].

    These are nice features. Sometimes, they are even useful (as opposed to just another hammer developers can abuse). But the announcement makes it seem, wrongly, that MSFT is doing something really unique here.

    • by spongman (182339)

      LOL!

      C# Has had the equivalent of javax.tools.JavaCompiler since v1.1, Microsoft.CSharp.Compiler [microsoft.com] which is little more than a wrapper around the command-line compiler.

      But the .NET framework has also, for a long time, included Reflection.Emit [microsoft.com] which allows for direct manipulation of CIL bytecode, and System.CodeDom [microsoft.com] for generating and compiling source code in multiple languages from an abstract representation.

      I wouldn't bother reading about Roslyn, though. It couldn't possibly measure up to anything Java has had

  • C# has always supported compiling additional code at runtime.

    I've had it in projects since the 1.0 release.

    They may be redoing the structure and making it easier to do, but doing it isn't new.

Old programmers never die, they just branch to a new address.

Working...