How FreeType's rasterizer work

                          by David Turner

                        Revised 2007-Feb-01

This file  is an  attempt to explain  the internals of  the FreeType

rasterizer.  The  rasterizer is of  quite general purpose  and could

easily be integrated into other programs.

  I. Introduction

 II. Rendering Technology

     1. Requirements

     2. Profiles and Spans

        a. Sweeping the Shape

        b. Decomposing Outlines into Profiles

        c. The Render Pool

        d. Computing Profiles Extents

        e. Computing Profiles Coordinates

        f. Sweeping and Sorting the Spans

I. Introduction

===============

  A  rasterizer is  a library  in charge  of converting  a vectorial

  representation of a shape  into a bitmap.  The FreeType rasterizer

  has  been  originally developed  to  render  the  glyphs found  in

  TrueType  files, made  up  of segments  and second-order  Béziers.

  Meanwhile it has been extended to render third-order Bézier curves

  also.   This  document  is   an  explanation  of  its  design  and

  implementation.

  While  these explanations start  from the  basics, a  knowledge of

  common rasterization techniques is assumed.

II. Rendering Technology

========================

1. Requirements

---------------

  We  assume that  all scaling,  rotating, hinting,  etc.,  has been

  already done.  The glyph is thus  described by a list of points in

  the device space.

  - All point coordinates  are in the 26.6 fixed  float format.  The

    used orientation is:

       ^ y

       |         reference orientation

       |

       *----> x

      0

    `26.6' means  that 26 bits  are used for  the integer part  of a

    value   and  6   bits  are   used  for   the   fractional  part.

    Consequently, the `distance'  between two neighbouring pixels is

    64 `units' (1 unit = 1/64th of a pixel).

    Note  that, for  the rasterizer,  pixel centers  are  located at

    integer   coordinates.   The   TrueType   bytecode  interpreter,

    however, assumes that  the lower left edge of  a pixel (which is

    taken  to be  a square  with  a length  of 1  unit) has  integer

    coordinates.

        ^ y                                        ^ y

        |                                          |

        |            (1,1)                         |      (0.5,0.5)

        +-----------+                        +-----+-----+

        |           |                        |     |     |

        |           |                        |     |     |

        |           |                        |     o-----+-----> x

        |           |                        |   (0,0)   |

        |           |                        |           |

        o-----------+-----> x                +-----------+

      (0,0)                             (-0.5,-0.5)

   TrueType bytecode interpreter          FreeType rasterizer

    A pixel line in the target bitmap is called a `scanline'.

  - A  glyph  is  usually  made  of several  contours,  also  called

    `outlines'.  A contour is simply a closed curve that delimits an

    outer or inner region of the glyph.  It is described by a series

    of successive points of the points table.

    Each point  of the glyph  has an associated flag  that indicates

    whether  it is  `on' or  `off' the  curve.  Two  successive `on'

    points indicate a line segment joining the two points.

    One `off' point amidst two `on' points indicates a second-degree

    (conic)  Bézier parametric  arc, defined  by these  three points

    (the `off' point being the  control point, and the `on' ones the

    start and end points).  Similarly, a third-degree (cubic) Bézier

    curve  is described  by four  points (two  `off'  control points

    between two `on' points).

    Finally,  for  second-order curves  only,  two successive  `off'

    points  forces the  rasterizer to  create, during  rendering, an

    `on'  point amidst them,  at their  exact middle.   This greatly

    facilitates the  definition of  successive Bézier arcs.

  The parametric form of a second-order Bézier curve is:

      P(t) = (1-t)^2*P1 + 2*t*(1-t)*P2 + t^2*P3

      (P1 and P3 are the end points, P2 the control point.)

  The parametric form of a third-order Bézier curve is:

      P(t) = (1-t)^3*P1 + 3*t*(1-t)^2*P2 + 3*t^2*(1-t)*P3 + t^3*P4

      (P1 and P4 are the end points, P2 and P3 the control points.)

  For both formulae, t is a real number in the range [0..1].

  Note  that the rasterizer  does not  use these  formulae directly.

  They exhibit,  however, one very  useful property of  Bézier arcs:

  Each  point of  the curve  is a  weighted average  of  the control

  points.

  As all weights  are positive and always sum up  to 1, whatever the

  value  of t,  each arc  point lies  within the  triangle (polygon)

  defined by the arc's three (four) control points.

  In  the following,  only second-order  curves are  discussed since

  rasterization of third-order curves is completely identical.

  Here some samples for second-order curves.

                                        *            # on curve

                                                     * off curve

                                     __---__

        #-__                      _--       -_

            --__                _-            -

                --__           #               \

                    --__                        #

                        -#

                                 Two `on' points

         Two `on' points       and one `off' point

                                  between them

                      *

        #            __      Two `on' points with two `off'

         \          -  -     points between them. The point

          \        /    \    marked `0' is the middle of the

           -      0      \   `off' points, and is a `virtual

            -_  _-       #   on' point where the curve passes.

              --             It does not appear in the point

              *              list.

2. Profiles and Spans

---------------------

  The following is a basic explanation of the _kind_ of computations

  made  by  the   rasterizer  to  build  a  bitmap   from  a  vector

  representation.  Note  that the actual  implementation is slightly

  different, due to performance tuning and other factors.

  However, the following ideas remain  in the same category, and are

  more convenient to understand.

  a. Sweeping the Shape

    The best way to fill a shape is to decompose it into a number of

    simple  horizontal segments,  then turn  them on  in  the target

    bitmap.  These segments are called `spans'.

                __---__

             _--       -_

           _-            -

          -               \

         /                 \

        /                   \

       |                     \

                __---__         Example: filling a shape

             _----------_                with spans.

           _--------------

          ----------------\

         /-----------------\    This is typically done from the top

        /                   \   to the bottom of the shape, in a

       |           |         \  movement called a `sweep'.

                   V

                __---__

             _----------_

           _--------------

          ----------------\

         /-----------------\

        /-------------------\

       |---------------------\

    In  order  to draw  a  span,  the  rasterizer must  compute  its

    coordinates, which  are simply the x coordinates  of the shape's

    contours, taken on the y scanlines.

                   /---/    |---|   Note that there are usually

                  /---/     |---|   several spans per scanline.

        |        /---/      |---|

        |       /---/_______|---|   When rendering this shape to the

        V      /----------------|   current scanline y, we must

              /-----------------|   compute the x values of the

           a /----|         |---|   points a, b, c, and d.

      - - - *     * - - - - *   * - - y -

           /     / b       c|   |d

                   /---/    |---|

                  /---/     |---|  And then turn on the spans a-b

                 /---/      |---|  and c-d.

                /---/_______|---|

               /----------------|

              /-----------------|

           a /----|         |---|

      - - - ####### - - - - ##### - - y -

           /     / b       c|   |d

  b. Decomposing Outlines into Profiles

    For  each  scanline during  the  sweep,  we  need the  following

    information:

    o The  number of  spans on  the current  scanline, given  by the

      number of  shape points  intersecting the scanline  (these are

      the points a, b, c, and d in the above example).

    o The x coordinates of these points.

    x coordinates are  computed before the sweep, in  a phase called

    `decomposition' which converts the glyph into *profiles*.

    Put it simply, a `profile'  is a contour's portion that can only

    be either ascending or descending,  i.e., it is monotonic in the

    vertical direction (we also say  y-monotonic).  There is no such

    thing as a horizontal profile, as we shall see.

    Here are a few examples:

      this square

                                          1         2

         ---->----     is made of two

        |         |                       |         |

        |         |       profiles        |         |

        ^         v                       ^    +    v

        |         |                       |         |

        |         |                       |         |

         ----<----

                                         up        down

      this triangle

             P2                             1          2

             |\        is made of two       |         \

          ^  | \  \                         |          \

          | |   \  \      profiles         |            \      |

         |  |    \  v                  ^   |             \     |

           |      \                    |  |         +     \    v

           |       \                   |  |                \

        P1 ---___   \                     ---___            \

                 ---_\                          ---_         \

             <--__     P3                   up           down

      A more general contour can be made of more than two profiles:

              __     ^

             /  |   /  ___          /    |

            /   |     /   |        /     |       /     |

           |    |    /   /    =>  |      v      /     /

           |    |   |   |         |      |     ^     |

        ^  |    |___|   |  |      ^   +  |  +  |  +  v

        |  |           |   v      |                 |

           |           |          |           up    |

           |___________|          |    down         |

                <--               up              down

    Successive  profiles are  always joined  by  horizontal segments

    that are not part of the profiles themselves.

    For  the  rasterizer,  a  profile  is  simply  an  *array*  that

    associates  one  horizontal *pixel*  coordinate  to each  bitmap

    *scanline*  crossed  by  the  contour's section  containing  the

    profile.  Note that profiles are *oriented* up or down along the

    glyph's original flow orientation.

    In other graphics libraries, profiles are also called `edges' or

    `edgelists'.

  c. The Render Pool

    FreeType  has been designed  to be  able to  run well  on _very_

    light systems, including embedded systems with very few memory.

    A render pool  will be allocated once; the  rasterizer uses this

    pool for all  its needs by managing this  memory directly in it.

    The  algorithms that are  used for  profile computation  make it

    possible to use  the pool as a simple  growing heap.  This means

    that this  memory management is  actually quite easy  and faster

    than any kind of malloc()/free() combination.

    Moreover,  we'll see  later that  the rasterizer  is  able, when

    dealing with profiles too large  and numerous to lie all at once

    in  the render  pool, to  immediately decompose  recursively the

    rendering process  into independent sub-tasks,  each taking less

    memory to be performed (see `sub-banding' below).

    The  render pool doesn't  need to  be large.   A 4KByte  pool is

    enough for nearly all renditions, though nearly 100% slower than

    a more comfortable 16KByte or 32KByte pool (that was tested with

    complex glyphs at sizes over 500 pixels).

  d. Computing Profiles Extents

    Remember that a profile is an array, associating a _scanline_ to

    the x pixel coordinate of its intersection with a contour.

    Though it's not exactly how the FreeType rasterizer works, it is

    convenient  to think  that  we need  a  profile's height  before

    allocating it in the pool and computing its coordinates.

    The profile's height  is the number of scanlines  crossed by the

    y-monotonic section of a contour.  We thus need to compute these

    sections from  the vectorial description.  In order  to do that,

    we are  obliged to compute all  (local and global)  y extrema of

    the glyph (minima and maxima).

           P2             For instance, this triangle has only

                          two y-extrema, which are simply

           |\

           | \               P2.y  as a vertical maximum

          |   \              P3.y  as a vertical minimum

          |    \

         |      \            P1.y is not a vertical extremum (though

         |       \           it is a horizontal minimum, which we

      P1 ---___   \          don't need).

               ---_\

                     P3

    Note  that the  extrema are  expressed  in pixel  units, not  in

    scanlines.   The triangle's  height  is certainly  (P3.y-P2.y+1)

    pixel  units,   but  its  profiles'  heights   are  computed  in

    scanlines.  The exact conversion is simple:

      - min scanline = FLOOR  ( min y )

      - max scanline = CEILING( max y )

    A problem  arises with Bézier  Arcs.  While a segment  is always

    necessarily y-monotonic (i.e.,  flat, ascending, or descending),

    which makes extrema computations easy,  the ascent of an arc can

    vary between its control points.

                          P2

                         *

                                       # on curve

                                       * off curve

                   __-x--_

                _--       -_

          P1  _-            -          A non y-monotonic Bézier arc.

             #               \

                              -        The arc goes from P1 to P3.

                               \

                                \  P3

                                 #

    We first  need to be  able to easily detect  non-monotonic arcs,

    according to  their control points.  I will  state here, without

    proof, that the monotony condition can be expressed as:

      P1.y <= P2.y <= P3.y   for an ever-ascending arc

      P1.y >= P2.y >= P3.y   for an ever-descending arc

    with the special case of

      P1.y = P2.y = P3.y     where the arc is said to be `flat'.

    As  you can  see, these  conditions can  be very  easily tested.

    They are, however, extremely important, as any arc that does not

    satisfy them necessarily contains an extremum.

    Note  also that  a monotonic  arc can  contain an  extremum too,

    which is then one of its `on' points:

        P1           P2

          #---__   *         P1P2P3 is ever-descending, but P1

                -_           is an y-extremum.

                  -

           ---_    \

               ->   \

                     \  P3

                      #

    Let's go back to our previous example:

                          P2

                         *

                                       # on curve

                                       * off curve

                   __-x--_

                _--       -_

          P1  _-            -          A non-y-monotonic Bézier arc.

             #               \

                              -        Here we have

                               \              P2.y >= P1.y &&

                                \  P3         P2.y >= P3.y      (!)

                                 #

    We need to  compute the vertical maximum of this  arc to be able

    to compute a profile's height (the point marked by an `x').  The

    arc's equation indicates that  a direct computation is possible,

    but  we rely  on a  different technique,  which use  will become

    apparent soon.

    Bézier  arcs have  the  special property  of  being very  easily

    decomposed into two sub-arcs,  which are themselves Bézier arcs.

    Moreover, it is easy to prove that there is at most one vertical

    extremum on  each Bézier arc (for  second-degree curves; similar

    conditions can be found for third-order arcs).

    For instance,  the following arc  P1P2P3 can be  decomposed into

    two sub-arcs Q1Q2Q3 and R1R2R3:

                    P2

                   *

                                    # on  curve

                                    * off curve

                                    original Bézier arc P1P2P3.

                __---__

             _--       --_

           _-             -_

          -                 -

         /                   \

        /                     \

       #                       #

     P1                         P3

                    P2

                   *

                   Q3                 Decomposed into two subarcs

          Q2                R2        Q1Q2Q3 and R1R2R3

            *   __-#-__   *

             _--       --_

           _-       R1    -_          Q1 = P1         R3 = P3

          -                 -         Q2 = (P1+P2)/2  R2 = (P2+P3)/2

         /                   \

        /                     \            Q3 = R1 = (Q2+R2)/2

       #                       #

     Q1                         R3    Note that Q2, R2, and Q3=R1

                                      are on a single line which is

                                      tangent to the curve.

    We have then decomposed  a non-y-monotonic Bézier curve into two

    smaller sub-arcs.  Note that in the above drawing, both sub-arcs

    are monotonic, and that the extremum is then Q3=R1.  However, in

    a  more general  case,  only  one sub-arc  is  guaranteed to  be

    monotonic.  Getting back to our former example:

                    Q2

                   *

                   __-x--_ R1

                _--       #_

          Q1  _-        Q3  -   R2

             #               \ *

                              -

                               \

                                \  R3

                                 #

    Here, we see that,  though Q1Q2Q3 is still non-monotonic, R1R2R3

    is ever  descending: We  thus know that  it doesn't  contain the

    extremum.  We can then re-subdivide Q1Q2Q3 into two sub-arcs and

    go  on recursively,  stopping  when we  encounter two  monotonic

    subarcs, or when the subarcs become simply too small.

    We  will finally  find  the vertical  extremum.   Note that  the

    iterative process of finding an extremum is called `flattening'.

  e. Computing Profiles Coordinates

    Once we have the height of each profile, we are able to allocate

    it in the render pool.   The next task is to compute coordinates

    for each scanline.