I'm trying to build a relational lang (in python, F#, Swift and now rust) and have stayed at the AST level all this time.
I only put the lexer to be done from someone that wanna learn rust.
----
Is incredible how hard is tu build more complex semantics than "1 + 1". I have even get some nice answers from the guy of crafting-interpreters and he not even now some of them!
Things you wanna know, and maybe you not find (easily) how:
* How incorporate AGDT and pattern matching in a lang. It take me A LOT of time to figure that you can encode it in arrays + labels (now you get more samples of do it... Is getting more popular with hobbits to implement it)
* How implement type inference
* How create generators, coroutines and the like, specially if your host not have it
* ok, add a debugger
* ok, add support for a IDE or at least a code editor
* How design a FFI system?
* How make a lang without GC and it work fine?
* ... and with a GC, not using the one in your host?
* Continuation passing style that perform well in a lang without call-cc
* How to test the lang, correctly! (have not even the idea yet)
* You read tons of paper with amazing ideas, that even claim to have worked. Then you find you can't read most of the "math stuff" inside and no code to look at them. Them PL guys get mad because we ignore them
> You read tons of paper with amazing ideas, that even claim to have worked.
They often do work, on a toy compiler that has a carefully selected subset of features expected in a professional compiler. Things like "let's not allow casting! or exceptions! or inline assembler! or interfacing to C! or integer wraparound!"
I know you have decades of experience with compilers, but are you sure 0.1% isn't a bit low? I can imagine 0.1% to be true if you have experience writing parsers, and just make a shitty parser for a batch compiler, and spend a lot of time on the semantics on the language.
You can also emit some shitty stack machine bytecode with a single AST pass, and write a simple interpreter for that bytecode, and the effort would be about the same as writing the simple parser by hand.
And you can spend a lot of time making a parser actually robust, threading file positions through the pipeline, handle errors nicely, allow for efficient incremental parsing, and so on. I could probably spend man-months and man-years on such a parser.
You write the parser once and then eventually forget about it except if there is a language change. Then you spend a small amount of time adding the syntax for that change to the parser and again forget about it.
On the other hand, _lots_ of folks work on the optimization passes, instruction selection, register allocation and the core of the IR. Since instruction selection and register allocation have CPU specific features usually, they have some logic whose complexity (in terms of size of the code) is proportional to the number of CPUs targeted.
So 0.1% is probably pretty close to the true number.
Inversely with LLVM around (and the JVM, sort of), people now get to focus on building the frontend of the compiler and forget about all those optimisation passes.
Only suckers forget about optimization passes. ;-)
Don't forget that:
- Optimization work on llvm and other compilers is still happening in anger.
- Frontend is a relative term. For example, the "frontend" to LLVM when WebKit used LLVM as a "backend" was a full-blown compiler with >100KLoC of code. Since then that "frontend" grew its own backend and dropped LLVM, but point is, just because you're a frontend doesn't mean you don't have optimizations. Frontend doesn't just mean parser.
- Rust's or Haskell's "frontends" to LLVM are seriously impressive compilers and the complexity is not to do with parsing AFAICT but all the other stuff (type checking comes to mind but there's also really impressive lowering and optimization).
- Backends aren't just complex because of optimization passes. The register allocator and instruction selector aren't so much optimization passes as necessary transformations for turning a compiler's IR into machine code. Those things just grow and grow.
- The whispers in the wind that I'm hearing from compiler folks looking to make bank is that there's shitloads of work to be done porting compilers to new targets. It's much more work to port a compiler to a new target than it is to write a parser. That said, writing a frontend for a language as complex as Rust/Haskell/etc is probably on the same order of complexity as parting a backend or maybe even more complex - but that's not because of parsing; it's usually either because of System F type shit (if you have a statically typed language) or dynamic type inference (if you have a dynamically typed language).
>now get to focus on building the frontend of the compiler
Although LLVM is cool, it's just a backend, so besides frontend you need some runtime, type checker, type inference, and, if your lang is sufficiently complex, you still need a huge middle end layer which knows something about semantics of your language and how to optimize it properly.
So if your language is just a very simple featureless language with procedures and simple data types, like Pascal, you could just write a direct code generator. But when you have a language with GC, exceptions, modules and all that kind of stuff, you need a middle end, and a complex one.
It's probably way too high. The reason is parsing is a small, simple, very well defined problem, and only rarely changes. This is quite unlike anything else in the compiler.
Unless, of course, you're talking about the C preprocessor. Prepare to spend an ungodly amount of time on that.
Yup, lexing and parsing are, by far, the easiest part of a compiler. Like about 0.1% of the work.