Artistic style transfer

July 10, 2017

In this experiment, we create a neural artistic style transfer model from a propaganda poster, and use it to demonstrate style transfer on some of our modern propaganda material.

The concept of artistic style transfer comes from efforts to visualize the states of trained machine learning models, especially neural networks.

It has been widely implemented, and the criterion we’re using to pick the first implementation to try out is the speed of execution.

Style transfer with Torch

The winner was https://github.com/jcjohnson/fast-neural-style or

@inproceedings{Johnson2016Perceptual,
  title={Perceptual losses for real-time style transfer and super-resolution},
  author={Johnson, Justin and Alahi, Alexandre and Fei-Fei, Li},
  booktitle={European Conference on Computer Vision},
  year={2016}
}

Our objective is to turn this propaganda poster into an artistic style transfer model:

poster

Installing Torch

We’re starting from http://torch.ch/docs/getting-started.html#installing-torch, which was written for GCC 4.9, rather than the GCC 6.3 shipping with Ubuntu 17.04 by default.

Not being a C/C++ developer, I don’t have a compiler toolset switcheroo script all ready.

So before we start, we’ll borrow some ideas from https://gist.github.com/esafak/534ad1a0e765dc8bee2251b9079e7f48 and push GCC behind the Ubuntu alternatives system.

$ sudo apt install build-essential gcc-4.9 g++-4.9

$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 15
update-alternatives: using /usr/bin/gcc-4.9 to provide /usr/bin/gcc (gcc) in auto mode
$ sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 15
update-alternatives: using /usr/bin/g++-4.9 to provide /usr/bin/g++ (g++) in auto mode
$ sudo update-alternatives --install /usr/bin/cc1 cc1 /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1 15
update-alternatives: using /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1 to provide /usr/bin/cc1 (cc1) in auto mode
$ sudo update-alternatives --install /usr/bin/cc1plus cc1plus /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1plus 15
update-alternatives: using /usr/lib/gcc/x86_64-linux-gnu/4.9/cc1plus to provide /usr/bin/cc1plus (cc1plus) in auto mode

$ gcc --version
gcc (Ubuntu 4.9.4-2ubuntu1) 4.9.4
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Then on with the show. Torch takes quite a while to compile…

If you accidentally started the compilation with GCC 6.3, and in the end, you’re getting crap results from the models in the instance_norm subdirectory, run ./clean.sh and ./install.sh to clean out incompatible object files.

After it’s done:

$ luarocks list
Warning: Failed loading manifest for /home/mikael/.luarocks/lib/luarocks/rocks: /home/mikael/.luarocks/lib/luarocks/rocks/manifest: No such file or directory

Installed rocks:
----------------

argcheck
   scm-1 (installed) - /home/mikael/devel/torch/install/lib/luarocks/rocks

cudnn
   scm-1 (installed) - /home/mikael/devel/torch/install/lib/luarocks/rocks

...

$ th
 
  ______             __   |  Torch7 
 /_  __/__  ________/ /   |  Scientific computing for Lua. 
  / / / _ \/ __/ __/ _ \  |  Type ? for help 
 /_/  \___/_/  \__/_//_/  |  https://github.com/torch 
                          |  http://torch.ch 
	
th> torch.Tensor{1,2,3}
 1
 2
 3
[torch.DoubleTensor of size 3]

While fast-neural-style’s installation instructions include re-installing various packages with luarocks, many of those packages were already built as a part of Torch, so unless you like wasting time, skip rebuilding packages already listed by luarocks list.

Validating the installation

Now let’s validate the installation. Clone fast-neural-style if you haven’t before, run the model download scripts as described in the repository’s README.md, and run:

th fast_neural_style.lua \
  -gpu 0 -backend cuda -use_cudnn 1 \
  -median_filter 2 -model models/instance_norm/mosaic.t7 \
  -input_image images/content/chicago.jpg -output_image /tmp/out.png \
  -image_size 1280

Now, /tmp/out.png should look like:

chicago-mosaic

If it instead looks like crap, you screwed up with your GCC versions - clean up and reinstall Torch.

Training your propaganda model

As described in the fast-neural-style documentation, first create a H5DF file out of the COCO dataset, the resulting file will be around 23 GB. I copied this file into /dev/shm to reduce the amount of network IO for ELB fetches on AWS, but on a local SSD, it won’t make much difference.

Let’s try with a piece of calligraphy from the ancient Chinese painting Nine Dragons:

Nine Dragons Piece


th train.lua \
   -h5_file /dev/shm/coco_model.h5 \
   -checkpoint_name chinese \
   -style_image ~/models/chinese.png -style_image_size 650 \
   -content_weights 1.0 -style_weights 5.0 \
   -gpu 0 -backend cuda -use_cudnn 1

...

Epoch 0.048321, Iteration 1000 / 40000, loss = 657568.825833    0.001   
Running on validation set ...   
val loss = 656873.192345

After a thousand iterations, the script creates a checkpoint which we can then try, to see where this training iteration is going.

The first checkpoint produced by the first training iteration one won’t be very indicative of the final result, but we can simply copy the created snapshot file chinese.t7 to a safe directory, and give it a shot.

ccfi

OK, maybe the final result will be sorta brownish?

Testing your propaganda model

(If you want to skip the training process, and go straight to using the propaganda model, you can download it from my Git repo.

th fast_neural_style.lua \
  -gpu 0 -backend cuda -use_cudnn 1 \
  -median_filter 2 -model models/mikael_propaganda.t7 \
  -input_image images/content/chicago.jpg -output_image /tmp/out.png \
  -image_size 1280

chicago-propaganda

Applied to my Twitter header image:

helsinki

helsinki-propaganda

Style transfer with Tensorflow

https://github.com/lengstrom/fast-style-transfer

https://www.moma.org/interactives/exhibitions/2012/inventingabstraction/?work=99

Provincetown

convert 99.jpg -strip -resize 512x512! marsden_hartley_provicetown_512x512.jpg

python style.py \
  --checkpoint-dir checkpoint/marsden_hartley_provincetown \
  --style marsden_hartley_provicetown_512x512.jpg \
  --train-path data/train2014 --content-weight 1.5e1 --checkpoint-iterations 1000 --batch-size 15 \
  --epochs 4