L

litbuild

a Literate Build system.

public

 
litbuild -- a literate build system
===================================

The idea of literate programming was invented by Donald Knuth. The basic
notion -- at least, the way I think of it -- is that the _primary_ users
of the source code of a program are the people who are reading the
source code to acquire an understanding of how it works (often, because
they want to modify it), and that the transformation of the source code
into an executable program that can be run, or the actual use of that
program, is a _secondary_ purpose of the code. Because of this
prioritization, most of the attention and energy invested in writing the
program is in a narrative discussion of how the program works and what
the purpose of each component (function declaration, subroutine or
coroutine, or whatever) is within that narrative.

Automated build tools -- such as make, maven, and jam -- are great,
because they allow developers on a project to focus their attention on
the project codebase _per se_, and let the build system worry about how
to compile and test it. There are lots of build tools, reflecting a wide
variety of design goals and value systems. Regardless of which tool is
used, though, I have noticed that once a software project reaches a
certain level of complexity, the automated build becomes similarly
complex -- to the point that it is a major undertaking to figure out how
it actually works.

That makes it difficult to diagnose and correct problems with the build
system when they arise, for the same reason that any other
incomprehensible software code is hard to work with effectively... and
since nobody pays any attention to the build machinery until it stops
working for some reason, I've also noticed that it's common for everyone
on a team or project to forget how it works in between those times.

Litbuild is an attempt to bring together the ideas of literate
programming and automated build tools. My goal is to make it easy to
write build scripts in such a way that they _tell a story_ about how to
build a project (regardless of how big or complex that project is),
focusing primarily on describing that build process with enough clarity
that other people can understand it -- while still providing enough
structure to that story that the litbuild system can produce scripts
that do an automated build of the project when desired.

The original literate programming systems, WEB and CWEB, have two modes
of operation: you can run the literate source code through a `tangle`
program to produce source code in a traditional programming language
like Pascal or C (which you can then feed to a compiler to produce a
program), or you can run it through a `weave` program that produces
document source (which you can send through a typesetting system like
TeX or Docbook).

Litbuild takes, as input, human-readable files -- referred to as
"blueprints" -- that consist of narrative text interspersed with command
blocks and other directives, just like the literate source in WEB and
CWEB. It has a simpler interface, though: when litbuild is run, it
always generates a set of bash scripts that automate the actual build
process and _also_ produces AsciiDoc-formatted documentation that can be
run through a document production toolchain to produce HTML or Docbook
or some other documentation format.

The vision of litbuild is that, from a set of blueprints, you should be
able to say:

"Right now, on this system, supposing that I want to have thus-and-so
package installed...

  * "Tell me a story about how I can set that up and have it working
    properly", or
  * "Produce a set of scripts that I can run to have that package set up
    and working properly."

Using Litbuild
==============

The litbuild gem provides a simple driver script, `lb`, that reads a
configuration file (by default .litbuildrc in the current directory),
loads all the blueprint files it finds in a directory tree, and then
generates output files for the blueprint target specified with the
`--target` command line parameter. You can specify more than one
blueprint, separated by commas, if you really want to; but it's probably
a better idea to create a Section blueprint that talks about all the
blueprints you want to build and how they fit together. If the target
you want to build is a phase within a blueprint, specify it as
`blueprint::phase_name`; if the phase has a space in it, you can of
course quote the command line argument like `"blueprint::phase name"`.

The configuration file, if you provide one, should contain just simple
`key: value` lines. If the default values for configuration parameters
are good enough, you don't need to worry about this!

To use the lb driver to build a complete cross-toolchain for a
Cross-Built Linux system, for example, you might use something like:

  lb -c myconfig -t gnu-cross-toolchain

It's not strictly _necessary_ to use Section blueprints to perform a
complete build for a complex target, because blueprints can itemize
their dependencies and litbuild will pull in whatever other blueprints
are needed to resolve those dependencies. If all dependencies are
correctly specified in all blueprints, all you really need to do is
request the final target! But it's still a good idea to write a Section
for any situation where the various components of a build story need to
have an overall framing narrative or discussion to make them
comprehensible, or when all the parts of a build need to share a common
configuration or environment -- which is the case for most complicated
processes!

Debugging Blueprints
====================

If litbuild is rejecting your blueprints with an error message and you
can't figure out why, you can produce a great deal of debugging
information by setting the environment variable RUBYOPT to `--debug`.

Cross-Built Linux
=================

The litbuild program is developed in conjunction with a set of litbuild
blueprints called "Cross-Built Linux," or CBL for short. A lot of the
features in litbuild are present for the convenience of CBL -- like the
s6-rc service directory and pipeline logic, the ability to produce
package-user scripts, and the invocation of configuration file
repository scripts. If you're using litbuild for some other purpose, and
don't want to make use of those features, you might want to eliminate
them.

We're thinking of eventually rewriting litbuild as a C program, to
eliminate its dependency on ruby; if we do that, we'll probably provide
a configuration option that allows features like these to be compiled
out of the program.