Artistic style transfer

July 10, 2017

In this experiment, we create a neural artistic style transfer model from a propaganda poster, and use it to demonstrate style transfer on some of our modern propaganda material.

The concept of artistic style transfer comes from efforts to visualize the states of trained machine learning models, especially neural networks.

It has been widely implemented, and the criterion we’re using to pick the first implementation to try out is the speed of execution.

Style transfer with Torch

The winner was or

  title={Perceptual losses for real-time style transfer and super-resolution},
  author={Johnson, Justin and Alahi, Alexandre and Fei-Fei, Li},
  booktitle={European Conference on Computer Vision},

Our objective is to turn this propaganda poster into an artistic style transfer model:


Installing Torch

We’re starting from, which was written for GCC 4.9, rather than the GCC 6.3 shipping with Ubuntu 17.04 by default.

Not being a C/C++ developer, I don’t have a compiler toolset switcheroo script all ready.

So before we start, we’ll borrow some ideas from and push GCC behind the Ubuntu alternatives system.

$ sudo apt install build-essential gcc-4.9 g++-4.9

$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 15
update-alternatives: using /usr/bin/gcc-4.9 to provide /usr/bin/gcc (gcc) in auto mode
$ sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 15
update-alternatives: using /usr/bin/g++-4.9 to provide /usr/bin/g++ (g++) in auto mode
$ sudo update-alternatives --install /usr/bin/cc1 cc1 /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1 15
update-alternatives: using /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1 to provide /usr/bin/cc1 (cc1) in auto mode
$ sudo update-alternatives --install /usr/bin/cc1plus cc1plus /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1plus 15
update-alternatives: using /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1plus to provide /usr/bin/cc1plus (cc1plus) in auto mode

$ gcc --version
gcc (Ubuntu 4.9.4-2ubuntu1) 4.9.4
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO

Then on with the show. Torch takes quite a while to compile…

If you accidentally started the compilation with GCC 6.3, and in the end, you’re getting crap results from the models in the instance_norm subdirectory, run ./ and ./ to clean out incompatible object files.

After it’s done:

$ luarocks list
Warning: Failed loading manifest for /home/mikael/.luarocks/lib/luarocks/rocks: /home/mikael/.luarocks/lib/luarocks/rocks/manifest: No such file or directory

Installed rocks:

   scm-1 (installed) - /home/mikael/devel/torch/install/lib/luarocks/rocks

   scm-1 (installed) - /home/mikael/devel/torch/install/lib/luarocks/rocks


$ th
  ______             __   |  Torch7 
 /_  __/__  ________/ /   |  Scientific computing for Lua. 
  / / / _ \/ __/ __/ _ \  |  Type ? for help 
 /_/  \___/_/  \__/_//_/  | 
th> torch.Tensor{1,2,3}
[torch.DoubleTensor of size 3]

While fast-neural-style’s installation instructions include re-installing various packages with luarocks, many of those packages were already built as a part of Torch, so unless you like wasting time, skip rebuilding packages already listed by luarocks list.

Validating the installation

Now let’s validate the installation. Clone fast-neural-style if you haven’t before, run the model download scripts as described in the repository’s, and run:

th fast_neural_style.lua \
  -gpu 0 -backend cuda -use_cudnn 1 \
  -median_filter 2 -model models/instance_norm/mosaic.t7 \
  -input_image images/content/chicago.jpg -output_image /tmp/out.png \
  -image_size 1280

Now, /tmp/out.png should look like:


If it instead looks like crap, you screwed up with your GCC versions - clean up and reinstall Torch.

Training your propaganda model

As described in the fast-neural-style documentation, first create a H5DF file out of the COCO dataset, the resulting file will be around 23 GB. I copied this file into /dev/shm to reduce the amount of network IO for ELB fetches on AWS, but on a local SSD, it won’t make much difference.

Let’s try with a piece of calligraphy from the ancient Chinese painting Nine Dragons:

Nine Dragons Piece

th train.lua \
   -h5_file /dev/shm/coco_model.h5 \
   -checkpoint_name chinese \
   -style_image ~/models/chinese.png -style_image_size 650 \
   -content_weights 1.0 -style_weights 5.0 \
   -gpu 0 -backend cuda -use_cudnn 1


Epoch 0.048321, Iteration 1000 / 40000, loss = 657568.825833    0.001   
Running on validation set ...   
val loss = 656873.192345

After a thousand iterations, the script creates a checkpoint which we can then try, to see where this training iteration is going.

The first checkpoint produced by the first training iteration one won’t be very indicative of the final result, but we can simply copy the created snapshot file chinese.t7 to a safe directory, and give it a shot.


OK, maybe the final result will be sorta brownish?

Testing your propaganda model

(If you want to skip the training process, and go straight to using the propaganda model, you can download it from my Git repo.

th fast_neural_style.lua \
  -gpu 0 -backend cuda -use_cudnn 1 \
  -median_filter 2 -model models/mikael_propaganda.t7 \
  -input_image images/content/chicago.jpg -output_image /tmp/out.png \
  -image_size 1280


Applied to my Twitter header image:



Style transfer with Tensorflow


convert 99.jpg -strip -resize 512x512! marsden_hartley_provicetown_512x512.jpg

python \
  --checkpoint-dir checkpoint/marsden_hartley_provincetown \
  --style marsden_hartley_provicetown_512x512.jpg \
  --train-path data/train2014 --content-weight 1.5e1 --checkpoint-iterations 1000 --batch-size 15 \
  --epochs 4