Making It Easier to Roll Your Own DSL on the JVM : Xtext 2.1 Now Available
Almost 10 years to the day, IBM open-sourced Eclipse, giving the developers a platform and IDE that has continually gone from strength to strength. In recent years, Xtext has been one of the most popular projects in the eco-system,providing the possibility to add your own DSL to Eclipse. The latest release provides a new API to build domain-specific languages for the JVM. I spoke with Sven Efftinge about what could be one of the most important steps that the Xtext project has taken.
DZone: Hi Sven, what are you releasing this week?
At EclipseCon Europe, we gave a preview of Xcore on Thursday at. This is a huge step for EMF, but it's just a preview for now. More on this during EclipseCon North America.
DZone: What are the key new features in this release?
Sven Efftinge: The release contains over 200 bug fixes and slightly more than 50 new features. At the same time it's binary compatible to Xtext 2.0.
Between all the good features there's one outstanding thing: Xtext 2.1 comes with a new API to build domain-specific languages for the JVM. The API is super easy and understandable but at the same time extremely powerful and it's totally amazing to see what comes out at the end.
DZone: What does support for JVM languages actually mean?
Sven Efftinge: In Xtext 1.0 we introduced the possibilities to refer to JVM elements, like types, members, etc. In Xtext 2.0 we introduced Xbase, a full-featured, statically typed expression language which you could embed into any languages built with Xtext. These expressions can be translated to Java code or can be executed by an interpreter running on the JVM.
That was already extremely cool and a big leap forward, but it was relatively complicated to integrate these expressions in your language. In addition to integrating the expressions in your grammar definition you had to customize all kinds of other language aspects like scoping, the type system, how names are computed and the compiler. This is has been dramatically simplified now!
In Xtext 2.1 there are only two small scripts you need to implement. And from that definition you get a fully working language, which is translated to executable Java code and the default tooling is close to what you are used to from Java's Development Tooling. Such a language can interact with Java types (including generics), can have expressions at arbitrary places and could even use full blown Java annotations where ever you like. It's completely up to you - the limitation is your imagination :-)
If you are on the Java platform and you work with Eclipse, working with a JVM-based DSL built with Xtext is an extremely low hanging fruit now.
The simplicity on the one hand and the powerfulness on the other is overwhelming. I'm curious what people will make out of it.
DZone: When implementing a DSL you mention two small scripts are required. What do these look like?
Sven Efftinge: The first script is the usual Xtext grammar, where you define the syntax of your language and how text is parsed into an in-memory model, such that further processing code can conveniently work with the parsed information.
The second script is the new one: It's basically a mapping from your language concepts to java concepts, that is classes, interfaces, enums and their members. You basically do a transformation from you model to the model of Java, but only for the coarse grained stuff, i.e. the signatures. Then you associate your Xbase expressions with a derived Java method, and by doing that you automatically define the scope, the type context and where and how to compile it. It might sound a bit complicated now, but once you got it is really simple and it is absolute amazing what comes out of it in the end.
The domain model example shipped with Xtext could no be reduced to just these two small text files. And its JVM model inferrer, that's what we call the second script, consists of 30 lines of code only. And that code is very readable and concise since we use Xtend's cool new builder API for that. I blogged about that recently, without explaining what we use it for :-)
DZone: Could you explain what makes Xtext so popular?
Sven Efftinge: On the one hand I think it's a mix of good ideas, a lot of passion and the courage to leave things out and say no to certain feature request.
Unfortunately, a lot of people think open-source means everybody should be able to add his or her features., but that's not how good products are made. So sticking to a vision and having one in the first place seems to be helpful.
On the other hand it's the good quality and the superior community support. We are serious about our code base, we do a lot of code reviews and have over 8000 unit tests.
Through the past three years my friend and partner in crime, Sebastian Zarnekow, taught me how important discipline is when you want to work on a sustainable long-living code base, which was one of the most important lessons learned in the past.
DZone: What are the major features on the Xtext roadmap?
Sven Efftinge: We still have a lot of ideas but we have to slow down a bit and put even more focus on stabilizing and improving what's already there.
The next big thing will be the book we just started to work on.