outline.txt 7.9 KB
High level outline of the book:

- Introduction

  What is this book?

  - Our primary goal: build, completely from source code, a
    complete and up-to-date GNU/Linux operating system
    installation for small devices that have at least a few
    gigabytes of storage but lack both direct human interaction
    I/O capabilities (keyboard and monitor) and usable serial
    ports.

  - Our secondary goal: make it easier for other people to do
    similar things in the future, if they have the need or
    inclination.

  - Our tertiary goal: impress women with our stunning technical
    prowess.  That works, right?  I think I saw it in xkcd.

  Is this book redundant?

  - Linux From Scratch and Do It Yourself Linux are similar, but
    have different focus (building desktop systems).  Neither is
    focused (are they? better check DIY) on clearly explaining
    the purpose of each individual step, and neither is very good
    at discussing the challenges of building on a device with no
    built-in console.

  - OpenWRT and other systems designed to build an embedded
    appliance are designed to build a small flash-resident image,
    rather than a full OS install, and are entirely built using a
    cross-toolchain.

  To whom might this book be valuable?

  - People who want to build a complete operating system
    distribution using free software, targeting a device that is
    not mainstream.

  - People who want to build a general-purpose, highly portable
    server that they can use for any arbitrary computing task.

  - People who would like to understand exactly what goes into an
    operating system installation, and how to produce one.

- Our Goal

  - A flash-resident image that contains the Linux kernel and
    enough supporting userspace code to initialize hardware and
    mount a real root filesystem (either on built-in storage or
    over NFS).

  - A full GNU/Linux operating system installation on the
    device's built-in storage.

  - A book describing the entire process we used to develop those
    things.
    - A description of our approach to discovering enough about
      the target device that we could put together a plan of
      attack.
    - A step-by-step explanation of the process of building the
      software, with an emphasis on why each step is important.

  - The development of scripts that make it relatively easy to
    build new systems as packages within the base system are
    upgraded.

- Device Discovery

  - We started by noticing the WL-700gE, a wireless router that
    has a built-in hard disk and power supply.
    - Gosh, that thing looks like a server except it's the size
      of a hardcover book!
    - It would be nice to use something like that as a firewall
    - Maybe we could use it for other stuff as well (web hosting,
      destination for backups, email, DNS, VOIP, ...)
    - It would be kind of nice to have a tiny little server so we
      could have a consistent computing environment no matter
      where we go.
    - What would it take to get a complete OS installation on
      that disk, instead of using the disk just for file storage?
      (Answer: a big pile of work.)

  - Finding device information on the Internet
    - Asus source tarball (has bad versions of software but very
      useful documentation)
    - OpenWRT
    - Asus forums
  - Hacking the device to have a usable serial console
    - Let's void the warranty!
  - Understanding the boot loader (CFE)
    - We can use CFE's recovery mode to flash a new firmware
      image
    - It would be great to use Linux as a second-stage boot
      loader, then have that copy of Linux boot the real runtime
      kernel from the hard disk or tftp or whatever
      - kexec (yay!)
      - kexec may not work for MIPS (boo!)
  - TFTP and NFS booting for feasability testing
  - Getting the hard disk to spin
    - OpenWRT "diag" kernel module and boot scripts
    - Make sure we can do it ourselves
  - What if you don't have an OpenWRT project to crib from? (you
    are targeting a different hardware device)
    - Learn things we didn't have to learn (become a kernel hacker)
    - Find a different target device

  - Identifying the constraints imposed by our target device
    - 2MB of flash, 1728kb available
    - flash must contain enough of a system to spin and mount the hard disk
    - ideally, we write to flash as few times as possible
    - First 256kb of flash is the CFE boot loader; without it, we
      have a brick.  (Don't cross the streams.)
    - No JTAG apparently means no way to un-brick the thing.
      (Really, don't cross the streams!)
    - no serial console available by default, let's try to ensure
      that a full build is possible without a serial console

  - Developing a plan of attack
    - The flash image will contain the kernel and enough of a
      root filesystem to get things running
    - Once the hardware is initialized, the kernel will mount the
      real root filesystem from the hard disk and pole-vault into
      it (chroot, pivot_root).
    - Tools needed to build the firmware: from OpenWRT; so we are
      good to go.
    - Operating system build: we can base that on CLFS.
      (Describe CLFS approach at a high level)
    - Build enough of a temporary system to get the thing into a
      state where we can finish the build
      - NFS-boot that temp system
      - Install the temp system to the internal HD and chroot
        into it to finish the build -- compiling over NFS is
        painful
    - Build-time firmware vs run-time firmware (another temporary
      component, like /cross-tools and /tools) -- just a kernel.
    - OS build a la CLFS
    - no console, so openssl and openssh are necessary (and
      telnetd for failsafe boot option)
    - in boot scripts: provide some kind of recovery failsafe
      option in case something is awry with sshd.  Maybe start a
      telnetd -- or use an external usb disk as the root fs
      location if there is one attached...
    - Runtime kernel, flash image tools, runtime flash image
    - Copy runtime flash elsewhere and use rescue mode to install
      it:  don't write to flash from linux-land, for fear of
      stepping on CFE.
    - Boot -- does any smoke emerge?
    - Profit!

- High-level description of the build process

  - Ensure the bootstrap system is adequate (list of build
    dependencies):
    - toolchain for building software
    - NFS server
    - trx image manipulation tools from OpenWRT
  - glibc-based cross-toolchain (A)
  - temporary system build tools (built using A); this is the
    initial part of CLFS through "if you are going to..." branch
    point
  - create a tarball (B) that includes the temporary system.
  - complete the "if you are going to chroot" branch; tar this up
    as well as (C).
  - restore the (B) tarball and complete the "if you are going to
    boot" branch.  This is the filesystem that will be
    NFS-mounted to complete the build.
    - additional requirements: fdisk, appropriate fs tools
      (reiser, xfs, e2fs, whatever), module-init-tools if not
      part of standard CLFS, vim
    - copy the (C) image into the NFS exported filesystem
  - set up NFS on bootstrap system; export the filesystem.
  - initial (bootstrap) flash image, containing:
    - Linux kernel with static network configuration and NFS root
      filesystem location hard-coded
    - kernel includes subset of OpenWRT patches, doesn't need
      mini_fo etc., needs lzma boot code etc
    - The kernel modules from that build go into the NFS root fs
  - Flash the bootstrap image using rescue mode
  - ALTERNATIVE: smaller subset of kernel patches, no LZMA,
    TFTP-boot kernel -- requires serial port on device.

  - Boot the freesa device using NFS root fs image
  - Activate the hard disk, partition, mkfs, mount, unpack (C) tarball
  - chroot into the partial system

  - Final system buildout as per CLFS with package users
  - Automated option

  - Boot scripts for normal system

  - build runtime flash image: flash filesystem boot scripts etc