Salient Painting


Works of art are salience maps, meaning that when anyone draws (no matter how skilled they  might be) the result is a map of the most important visual elements needed to understand a scene. Without this artists (skill notwithstanding) would not be able to draw at all, instead they would record all parts of the scene without discrimination - which is exactly what photographs do. Moreover, the salience of a visual element is not an inherent property of the object, rather salience depends on all other objects in the scene - it is a relative, non-local property.

This page shows results of painting using filters, in which small patches of an image are replaced with a mark that affects a paint-brush stroke. This filtering approach is the basis of much of nonphotorealistic rendering from photographs, but we were first to recognise and use global salience as a control. We went on to build a filtering system based that not only includes global salience but which can be trained by users to recognise particular patterns (user selected)  such as corners and edges. Finally this was integrated in the first system that used genetic search to determine where to lay down strokes.

    J. Collomosse and P. Hall, “Painterly rendering using image salience”,
    Eurographics UK, 122-128, 2002.

    P.Hall and M. Owen, “A trainable low-level feature detector”,
    International Conference on Pattern Recognition, 708-711, 2004.

    J. Collomosse and P. Hall, “Genetic Paint: A Search for Salient Paintings”
Lecture Notes in Computer Science (Proc. EvoMUSART), vol. 3449, pp. 437-447, 2005

A global salience measure improves the quality of NPR.  Above, from left to right: A rendering using a local edge map as a guide to lay down strokes; stroke positions decided by global salience but with stroke properties determined from an edge map; right stroke position and form both determined by global salience.

Right, global salience maps compared to local edge maps. Global salience picks out a circle that edge maps fail to detect. It also identifies visual objects by colour and orientation. In each case it is closer to ground truth than the edge map.

We can extend global salience using a classifier that has been trained by a human. The classifier identifies whether a salient region is an edge, a corner, or a ridge. Below left shows the result of applying the trained filter to a photograph of houses; corners are coded in red, edges in blue and ridges in green. Below right shows a painting in which the form of stroke is determined by the class of pattern. Notice this allows the differential emphasis of objects.

Further training (below left) produces a scale-invariant detector.  A genetic algorithm uses this to decide where strokes should be, see how the strokes converge to details (middle). The differential emphasis of visual elements is clear, below right; the face is rendered with satisfactory detail whereas the rock surface is blurred away.