[Flang][driver] Update the docs

Please note that the updated documentation reflects the design that we are
working towards, but haven't implemented competely yet. We took this approach
in order to provide a more holisitic and complete overview of the design.

In particular, this document assumes that Flang's frontend and compiler
driver can already generate code. This is still work-in-progress and is
being developed on the `fir-dev` branch in the F18 repository [1].

[1] https://github.com/flang-compiler/f18-llvm-project/tree/fir-dev

Differential Revision: https://reviews.llvm.org/D111573

flang/docs/FlangDriver.md

@@ -13,24 +13,107 @@
    :local:
 ```
 
+
+> **_NOTE:_** This document assumes that Flang's drivers can already generate code and
+> produce executables. However, this is still work-in-progress. By making this
+> assumption, we are able to prepare this document ahead-of-time and to provide
+> an overview of the design that we are working towards.
+
 There are two main drivers in Flang:
 * the compiler driver, `flang-new`
 * the frontend driver, `flang-new -fc1`
 
-The compiler driver will allow you to control all compilation phases (i.e.
+> **_NOTE:_** The diagrams in this document refer to `flang` as opposed to
-preprocessing, frontend code-generation, middlend/backend code-optimisation and
+> `flang-new`. This is because the diagrams reflect the final design that we
-lowering, linking). For frontend specific tasks, the compiler driver creates a
+> are still working towards. See the note on [the flang script](https://github.com/llvm/llvm-project/blob/main/flang/docs/FlangDriver.md#the-flang-script)
-Fortran compilation job and delegates it to `flang-new -fc1`, the frontend driver.
+> below for more context.
 
-The frontend driver glues all of the frontend libraries together and provides
+The **compiler driver** will allow you to control all compilation phases (e.g.
-an easy-to-use and intuitive interface to the frontend. It accepts many
+preprocessing, semantic checks, code-generation, code-optimisation, lowering
-frontend-specific options not available in `flang-new` and as such it provides a
+and linking). For frontend specific tasks, the compiler driver creates a
-finer control over the frontend. Similarly to `-Xclang` in `clang`, you can use
+Fortran compilation job and delegates it to `flang-new -fc1`, the frontend
-`-Xflang` to forward the frontend specific flags from the compiler directly to
+driver. For linking, it creates a linker job and calls an external linker (e.g.
-the frontend driver.
+LLVM's [`lld`](https://lld.llvm.org/)). It can also call other tools such as
+external assemblers (e.g. [`as`](https://www.gnu.org/software/binutils/)). In
+Clang, the compiler driver can also link the generated binaries with LLVM's
+static analysis/sanitizer libraries (e.g.
+[MemorySanitizer](https://clang.llvm.org/docs/MemorySanitizer.html)). This is
+not yet available in Flang, but will be relatively easy to support once such
+libraries become available. Flang's compiler driver is intended for Flang's
+end-users - its interface needs to remain stable. Otherwise, Flang's users will
+have to adjust their build scripts every time a compiler flag is changed.
+
+| ![Compiler Driver](compiler_driver.png) |
+|:--:|
+| *Flang’s compiler driver and the **tools** that it runs* |
+
+The **frontend driver** glues together and drives all of the Flang's frontend
+libraries. As such, it provides an easy-to-use and intuitive interface to the
+frontend. It uses MLIR and LLVM for code-generation and can be viewed as a
+driver for Flang, LLVM and MLIR libraries. Contrary to the compiler driver, it
+is not capable of calling any external tools (including linkers).  It is aware
+of all the frontend internals that are "hidden" from the compiler driver. It
+accepts many frontend-specific options not available in `flang-new` and as such
+it provides a finer control over the frontend. Note that this tool is mostly
+intended for Flang developers. In particular, there are no guarantees about the
+stability of its interface and compiler developers can use it to experiment
+with new flags.
+
+| ![Frontend Driver](frontend_driver.png) |
+|:-:|
+| *Flang's frontend driver and the **libraries** that it drives* |
+
+Note that similarly to `-Xclang` in `clang`, you can use `-Xflang` to forward a
+frontend specific flag from the _compiler_ directly to the _frontend_ driver,
+e.g.:
+
+```lang=bash
+flang-new -Xflang -fdebug-dump-parse-tree input.f95
+```
+
+In the invocation above, `-fdebug-dump-parse-tree` is forwarded to `flang-new
+-fc1`. Without the forwarding flag, `-Xflang`, you would see the following
+warning:
+
+```lang=bash
+flang-new: warning: argument unused during compilation:
+```
+
+As `-fdebug-dump-parse-tree` is only supported by `flang-new -fc1`, `flang-new`
+will ignore it when used without `Xflang`.
+
+## Why Do We Need Two Drivers?
+As hinted above, `flang-new` and `flang-new -fc1` are two separate tools. The
+fact that these tools are accessed through one binary, `flang-new`, is just an
+implementation detail. Each tool has a separate list of options, albeit defined
+in the same file: `clang/include/clang/Driver/Options.td`.
+
+The separation helps us split various tasks and allows us to implement more
+specialised tools. In particular, `flang-new` is not aware of various
+compilation phases within the frontend (e.g. scanning, parsing or semantic
+checks). It does not have to be. Conversely, the frontend driver, `flang-new
+-fc1`, needs not to be concerned with linkers or other external tools like
+assemblers. Nor does it need to know where to look for various systems
+libraries, which is usually OS and platform specific.
+
+One helpful way of differentiating these tools is to keep in mind that:
+
+* the compiler driver is an end-user tool
+* frontend driver is a compiler developer tool with many additional options,
+
+Also, Since the compiler driver can call external tools, e.g. linkers, it can
+be used to generate **executables**. The frontend driver cannot call external
+tools and hence can only generate **object files**. A similar model is
+implemented in Clang (`clang` vs `clang -cc1` vs `clang -cc1as`), which is
+based on the [architecture of
+GCC](https://en.wikibooks.org/wiki/GNU_C_Compiler_Internals/GNU_C_Compiler_Architecture).
+In fact, Flang needs to adhere to this model in order to be able to re-use
+Clang's driver library. If you are more familiar with the [architecture of
+GFortran](https://gcc.gnu.org/onlinedocs/gcc-4.7.4/gfortran/About-GNU-Fortran.html)
+than Clang, then `flang-new` corresponds to `gfortran` and `flang-new -fc1` to
+`f951`.
 
 ## Compiler Driver
-
 The main entry point for Flang's compiler driver is implemented in
 `flang/tools/flang-driver/driver.cpp`.  Flang's compiler driver is implemented
 in terms of Clang's driver library, `clangDriver`. This approach allows us to:
@@ -92,9 +175,9 @@ You can read more on the design of `clangDriver` in Clang's [Driver Design &
 Internals](https://clang.llvm.org/docs/DriverInternals.html).
 
 ## Frontend Driver
-Flang's frontend driver is the main interface between end-users and the Flang
+Flang's frontend driver is the main interface between compiler developers and
-frontend. The high-level design is similar to Clang's frontend driver, `clang
+the Flang frontend. The high-level design is similar to Clang's frontend
--cc1` and consists of the following classes:
+driver, `clang -cc1` and consists of the following classes:
 * `CompilerInstance`, which is a helper class that encapsulates and manages
   various objects that are always required by the frontend (e.g. `AllSources`,
   `AllCookedSources, `Parsing`, `CompilerInvocation`, etc.). In most cases