Pix2PixHD is an image-to-image translation model, which synthesizes an input image A into an output image B. In order too simulate line drawings, canny edge detection was applied to the Yurukyara 2 dataset to create trainingset A with trainingset B, the 'target' the model learns to reproduce, consisting of the original images.
Some of the technical limitations are the canvas size of 512 by 512 pixels and the fact that Pix2Pix doesn’t support transparency. Additionally because of the use of canny edge detection, which attempts to convert all the edges of different colored shapes into lines, images of trainingset A are incredibly detailed and not necessarily true to how a human would draw. Thus one has to draw the input very cleanly with thin lines to achieve a reliable output.
Below you can find some drawing tests and 2D frame by frame animation tests with both inputs and outputs.