a Literate Build system.


litbuild -- a literate build system

The idea of literate programming was invented by Donald Knuth. The basic
notion -- at least, the way I think of it -- is that the _primary_ users
of the source code of a program are the people who are reading the
source code to acquire an understanding of how it works (often, because
they want to modify it), and that the transformation of the source code
into an executable program that can be run, or the actual use of that
program, is a _secondary_ purpose of the code.

Because of this prioritization, most of the attention and energy
invested in writing the program is in a narrative discussion of how the
program works and what the purpose of each component (function
declaration, subroutine or coroutine, or whatever) is within that

Automated build tools -- such as make, maven, and jam -- are great,
because they allow developers on a project to focus their attention on
the project codebase _per se,_ and let the build system worry about how
to compile and test it. There are lots of build tools, reflecting a wide
variety of design goals and value systems. Regardless of which tool is
used, though, I have noticed that once a software project reaches a
certain level of complexity, the automated build becomes similarly
complex -- to the point that it is a major undertaking to figure out how
it actually works.

That makes it difficult to diagnose and correct problems with the build
system when they arise, for the same reason that any other
incomprehensible software code is hard to work with effectively... and
since nobody pays any attention to the build machinery until it stops
working for some reason, I've also noticed that it's common for everyone
on a team or project to forget how it works in between those times. At
best, there's one person who knows how it all works, and he or she gets
saddled with the job of fixing it or tweaking it when that's necessary.

Litbuild is an attempt to bring together the ideas of literate
programming and automated build tools. My goal is to make it easy to
write build scripts in such a way that they _tell a story_ about how to
build a project (regardless of how big or complex that project is),
focusing primarily on describing that build process with enough clarity
that other people can understand it -- while still providing enough
structure to that story that the litbuild system can produce scripts
that will perform an automated build of the project.

The original literate programming systems, WEB and CWEB, have two modes
of operation: you can run the literate source code through a `tangle`
program to produce source code in a traditional programming language
like Pascal or C (which you can then feed to a compiler to produce a
program), or you can run it through a `weave` program that produces
document source (which you can send through a typesetting system like
TeX or Docbook).

Litbuild is essentially a processor for a domain-specific language. It
takes, as input, human-readable files -- referred to as "blueprints" --
that consist of narrative text written with AsciiDoc markup,
interspersed with command blocks and other directives, just like the
literate source in WEB and CWEB. Litbuild has a simpler interface,
though: when it is run, it always generates both a set of bash scripts
that will execute the actual build process and _also_ AsciiDoc-formatted
documentation files that can be run through a document production
toolchain to produce HTML or PDF or whatever other documentation format
you wish.

The vision of litbuild is that, from a set of blueprints, you should be
able to say:

"Right now, on this system, supposing that I want to have thus-and-so
package installed...

  * "Tell me a story about how I can set that up and have it working
    properly", or
  * "Produce a set of scripts that I can run to have that package set up
    and working properly,"

Or both!

Using Litbuild

The litbuild gem provides a simple driver script, `lb`, that loads all
the blueprint files it finds in a directory tree and then generates
output files for the blueprint target specified on the command line. You
can only specify one blueprint as a target; if you want more than one
thing to be built, create a Section blueprint that talks about all the
blueprints you want to build and how they fit together. If the target
you want to build is a phase within a blueprint, specify it as
`blueprint::phase_name`; if the phase has a space in it, you can of
course quote the command line argument like `"blueprint::phase name"`.

Blueprints can refer to parameterized values. It is expected that
default values for all these parameters will be provided (by convention,
I generally put all the default values into a "configuration" narrative
blueprint). To override any parameter value from the default, set an
environment variable with its name before running the `lb` driver

To use the lb driver to build a complete cross-toolchain as part of the
Cross-Building Linux process, for example, you might use something like:

  lb gnu-cross-toolchain

It's not strictly _necessary_ to use Section blueprints to perform a
complete build for a complex target, because blueprints can specify what
other bleuprints they depend on and litbuild will pull in whatever other
blueprints are needed to resolve those dependencies. If all dependencies
are correctly specified in all blueprints, all you really need to do is
request the final target! But it's still a good idea to write a Section
for any situation where the various components of a build story need to
have an overall framing narrative or discussion to make them
comprehensible, or when all the parts of a build need to share a common
configuration or environment -- which is the case for most complicated

If any commands in the generated scripts appear to be run under `sudo`,
litbuild has a feature that can help to run those scripts
non-interactively: it will produce sudoers entries (which can be
included in `/etc/sudoers` or in a file in `/etc/sudoers.d`) that permit
exactly those commands to be run under sudo without a password. For this
feature to work, all programs run under `sudo` must either be specified
with a full absolute path, or must exist in a specific set of default
directories: /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/bin, and

Debugging Blueprints

If litbuild is rejecting your blueprints with an error message and you
can't figure out why, you can produce a great deal of debugging
information by setting the environment variable RUBYOPT to `--debug`.

Cross-Building Linux

The litbuild program is developed in conjunction with a set of
blueprints called "Cross-Building Linux," or CBL for short; this defines
the build process used for Little Blue Linux. A lot of the features in
litbuild are present for the convenience of CBL -- like the s6-rc
service directory and pipeline logic, the ability to produce
package-user scripts, and the invocation of configuration file
repository scripts. If you're using litbuild for some other purpose, and
don't want to make use of those features, you might want to eliminate

We're thinking of eventually rewriting litbuild as a C program to
eliminate its dependency on ruby; if we do that, we'll probably provide
a compile-time configuration option that allows features like these to
be compiled out of the program.