Bringing Ghibli Magic to Photos with Generative AI

By Yewon Park

Photo by Afif Ramdhasuma on Unsplash

I’m obsessed with Studio Ghibli films. The soft textures, whimsical scenes, and painterly colors make me wish real life looked more like a Ghibli movie.

So for my MSDS 631 final project, I thought:

“Hey, why not use AI to transform real photos into Ghibli-style art?”

Turns out… that’s a bit ambitious. Especially when you only have 140 training images.

Here’s what happened.

Why Ghibli?

Studio Ghibli’s style isn’t just pretty. It’s distinctive:

• Soft, watercolor-like shading
• Dreamy, magical vibes
• Subtle, delicate color palettes
• Gorgeous fine details

It’s basically the perfect test of whether generative AI can capture more than just color: can it learn style?

I wanted to see if I could teach a model to translate ordinary photos into scenes that look like they came straight from Ghibli’s animation desk.

The Dataset

Fortunately, there’s a Kaggle dataset called the Ghibli Art Image Dataset. It contains 140 folders. Each folder has:

• o.png → an original real-world photo
• g.png → the same scene stylized in Ghibli’s look

One thing I discovered: the Kaggle splits for training, validation, and testing overlap the same image pairs (e.g. folder “1” exists in all splits). To avoid accidentally training and testing on the same data, I decided to just use the full set of 140 unique image pairs as my training data.

Why this dataset was perfect:

✅ It’s paired → ideal for supervised learning
✅ It’s small enough to fit on my laptop or run on Colab
✅ It’s focused exactly on my problem: Ghibli style transfer

Example image pair from the dataset

Preprocessing

Before I could train anything, I had to prepare the data.

• Resizing & Cropping → scaled all images to 256×256 pixels.
• Normalization → scaled pixel values between -1 and 1 to stabilize training.
• Augmentation → applied random flips and color jitter to help avoid overfitting.
• Data Splitting → divided the data into:
• 80% for training (112 pairs)
• 10% for validation (14 pairs)
• 10% for testing (15 pairs)

The First Attempt: U-Net

My first model was a U-Net, a popular architecture for image-to-image tasks. It works like this:

• An encoder compresses the input image into features.
• A decoder reconstructs a new image from those features.

How It Went

• Good: The model captured basic colors and rough shapes.
• Bad: The outputs were blurry, lacking artistic details or fine textures.

This happens because pixel-level loss functions (like MSE) tend to produce averages of possible outputs — great for structure, terrible for style.

U-Net first attempt sample output

Trying to Improve U-Net

I made several tweaks:

• Switched to L1 loss instead of MSE to reduce blurring.
• Added perceptual loss using a pre-trained VGG-16 network to capture style and textures.
• Tried data augmentation to help the model generalize better.

Sadly, these changes were still far from looking like real Ghibli art.

Improved U-Net Sample Output

Next Up: Pix2Pix

Next, I tried Pix2Pix, a type of Generative Adversarial Network (GAN) designed for paired image translation.

Get Ypark’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

How it works:

• The Generator creates a Ghibli-style image from a photo.
• The Discriminator judges whether that output looks real compared to actual Ghibli images.
• Training is a tug-of-war:
• The generator tries to fool the discriminator.
• The discriminator tries to spot fakes.

I combined:

• L1 loss → keeps the generated image similar to the original.
• Adversarial loss → pushes the generator toward realism.
• Perceptual loss → helps capture stylistic details.

The Challenges

Pix2Pix definitely introduced some improvements:

• Colors looked more vibrant.
• Broad shapes matched the original photos.

But…

• Outputs were still blurry.
• Fine textures were missing.
• Grid-like artifacts appeared, a common issue in GANs using transposed convolutions.

My dataset was simply too small for the model to fully learn the subtle, painterly effects that make Ghibli’s art so magical.

Pix2Pix best model sample output

The Results

Here’s how the models performed:

U-Net

• Best Validation Loss: 0.2314
• Average Test Loss: 0.2175

Pix2Pix

• Final Generator Loss: 4.4613
• Final Discriminator Loss: 0.6605

Quantitative losses aside, the outputs still didn’t truly look Ghibli-like.

What I Learned

• Supervised generative models can handle basic color and shape transfer — but they struggle with high-level artistic details on small datasets.
• Perceptual loss helps capture textures but needs more training data to shine.
• GANs introduce sharper outputs but also artifacts if under-trained.
• For detailed artistic styles like Ghibli, diffusion models might be a better choice.

Next Steps

If I continue this project, I’d love to:

✅ Expand the dataset — either with more images or synthetic data.
✅ Experiment with diffusion models like Stable Diffusion’s img2img for style transfer.
✅ Tune perceptual loss weights for better detail capture.
✅ Try pre-trained style models to save time and improve quality.

Final Thoughts

Creating Ghibli-style art with AI turned out to be far more challenging than I expected. While my models could match colors and general shapes, they fell short of capturing the soft, magical textures that make Ghibli movies so captivating.

But I learned a lot about generative AI — and I’m excited to keep exploring how technology and art can come together.

Thanks for reading my journey! Let me know if you’ve tried style transfer or have ideas for pushing this further.