A Plugin for Krita: Unleashing Artistic Potential with Generative AI

Krita, the renowned free and open-source painting program, takes a leap forward with the Generative AI plugin. This powerful tool integrates AI image generation into your Krita workflow, empowering you to create and refine artwork in innovative ways. It allows you to manipulate specific areas or your entire canvas using AI, producing new content that seamlessly blends with your existing artwork, like effortlessly removing unwanted elements, adding intricate details, or expanding your canvas with the AI replicating the style and theme.

Good! The introduction is finished… More details in:

1. Key Features: Unleashing Artistic Potential
2. Unveiling the Technical Aspects
3. Mastering Inpainting: Effortlessly Modify Your Artwork
4. Beyond Inpainting: Exploring Other Creative Options
5. Conclusion

Key Features: Unleashing Artistic Potential

The plugin boasts a diverse set of features designed to streamline your creative process:

Effortless Image Generation: Gone are the complexities of independent AI art creation. Simply define an area (or your entire canvas) and let the plugin generate new content based on your existing image or a text prompt (optional).

Multiple Creation Options: The plugin offers a variety of functionalities to suit your artistic needs:

• Inpaint: This mode allows you to seamlessly remove unwanted elements or replace specific sections of your image.
• Outpaint: Extend your canvas and automatically fill the new space with AI-generated content that continues the existing artwork’s style and theme.
• Generate: Create entirely new images based on your descriptions or existing images.
• Refine: Subtly adjust existing content or introduce new elements with a touch of finesse. Simply paint a rough approximation and use AI refinement for polished results.
• Live Painting: Get real-time feedback as you paint! The AI interprets your canvas and suggests continuations, fostering an interactive creative process.

Advanced Control and Customization: Experienced users can leverage the plugin’s extensive control features:

• Control Image Creation: Guide the generation process using additional information like sketches, line art, depth maps, or even character pose transfers.
• Resolution Flexibility: Work efficiently at any resolution. The plugin automatically adjusts AI model requirements and scales results to fit your image area.
• Upscaling Magic: Breathe new life into your images by upscaling them to high resolutions (4k, 8k+) without sacrificing quality.
• Job Queue and History: Manage generation tasks efficiently with a job queue. Additionally, browse past generations and prompts for easy reference.
• Presets and Customization: Utilize versatile default presets or create your own. Select specific Stable Diffusion checkpoints, add custom LoRA models, and fine-tune various settings for personalized results.

Unveiling the Technical Aspects

While the plugin offers a user-friendly interface, to use it you need the following:

System Requirements:

• Operating System: Windows, Linux, MacOS
• Hardware: Powerful graphics card with at least 6 GB VRAM is highly recommended for smooth operation.
• Supported GPUs are: NVIDIA (recommended via CUDA), AMD (supported via DirectML on Windows, ROCm on Linux — custom server only), Apple M1/M2 (supported via MPS on macOS). Also CPU processing is supported but significantly slower.
• Python (version 3.11 or 3.10 recommended) must be installed.

Integration with Krita:

1. The plugin requires Krita version 5.2.0 or newer.
2. Download and install the plugin.
3. Within Krita, navigate to Settings > Dockers > AI Image Generation to activate the plugin docker.
4. Configure the plugin by starting the local server installation or connecting to an existing one.
5. Restart Krita.

Backend and Model Support:

• The plugin utilizes ComfyUI as its backend. You can install it manually or use an existing installation. If the server is already running locally before starting Krita, the plugin will automatically connect. Remote server connections are also possible.
• Ensure compatibility by checking the list of required extensions and models.

Mastering Inpainting: Effortlessly Modify Your Artwork

Inpainting is a core functionality of the Generative AI plugin, allowing you to seamlessly modify specific areas of your image:

Get Elmo’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Selection is Key: Define the area you want to modify using Krita’s selection tools. This could be a small object you want to remove or a larger section you wish to replace.

Choosing the Right Mode: The plugin automatically adjusts the “Generate” button based on your selection. Here’s a breakdown of the common modes and their functionalities:

• Fill (Default): This versatile mode excels at general inpainting tasks. It replaces the selected area with content that resembles the surrounding area while offering some creative freedom for variations.
• Expand: Perfect for extending your canvas seamlessly. Choose a direction (left, right, top, or bottom) and the plugin will fill the new space with imagery that continues the style and theme of your existing artwork.
• Add Content: This mode is ideal for introducing entirely new elements into your image. Combine it with text prompts or control layers to describe the desired content for the AI to generate.
• Remove Content: This mode focuses on object removal. It fills the selected area with a continuation of the background, essentially erasing the unwanted element.
• Replace Background (Experimental): This mode attempts to replace the entire background behind a selected foreground object. It requires text prompts to describe the new background but might not always produce perfect results due to training limitations on the underlying AI model.

Customizing Inpainting: The plugin offers several options to fine-tune the inpainting process:

• Seamless (Enabled by Default): This option ensures the generated content blends seamlessly with the existing image. Disable it if you’re working on a specific panel within a larger artwork, like a comic page, where a clear separation might be desirable.
• Focus: By enabling this option, you can focus your text prompts solely on the selected area. Otherwise, your prompt will influence the entire context area surrounding the selection.
• Fill: This setting determines how the selected area is prepared before generation, impacting the results. Options include: None (Leaves the selected area untouched for further manual painting after inpainting), Neutral (Allows for unbiased generation without favoring specific colors in the surrounding area), Blur (Blurs the selected area and its surroundings to create a smoother transition for the generated content), Border (Ignores the existing content within the selection and prioritizes colors from the surrounding area for a cleaner blend), Inpaint (Fills the selected area using a specialized AI model specifically trained for inpainting tasks, often leading to more natural-looking results).

Context Area: A portion of the image surrounding your selection is used as context for the AI generation. This helps ensure the generated content integrates coherently with the existing artwork. A larger context area leads to better results but requires more processing power and memory. The plugin usually selects an appropriate context area automatically, but you can adjust it by changing Settings > Diffusion > Selection Padding.

Advanced Techniques: Working with Selection Masks

For even more control over the context used for inpainting, you can leverage selection masks:

1. Ensure “Global Selection Mask” is enabled in Krita’s settings.
2. Use the rectangular selection tool to mark your desired context area. This could be a specific area you want the AI to prioritize when generating content for the inpainted selection.
3. Save this selection mask for future use by clicking the dashed circle icon and assigning it a name.
4. Create a separate selection for the area you actually want to modify (e.g., poorly generated hands).
5. Within the custom generation menu, choose your saved selection mask as the context for the inpainting process.

This workflow offers full customization over the context used for generating content within your selected area, allowing you to guide the AI towards a specific outcome.

Beyond Inpainting: Exploring Other Creative Options

The Generative AI plugin extends far beyond inpainting, offering a plethora of creative possibilities:

• Outpainting: Expand your canvas and seamlessly fill the new space with AI-generated content that matches the style and theme of your existing artwork. This is perfect for creating expansive landscapes or adding details to the edges of your composition.
• Image Generation: Unleash your creativity by generating entirely new images based on your descriptions or existing images as a starting point. Text prompts play a crucial role here, allowing you to guide the AI towards your artistic vision.
• Refine and Enhance: Don’t discard elements you’ve already painted! The plugin allows you to subtly adjust existing content or add new details. Simply paint a rough approximation of the desired changes and use AI refinement to achieve a polished look. This can be particularly useful for adding intricate details or fixing minor imperfections.
• Live Painting: Embrace an interactive creative process with live painting. As you paint on your canvas, the AI interprets your strokes and suggests continuations in real-time. This allows you to experiment with different directions and receive immediate feedback from the AI, fostering a more fluid and dynamic workflow.
• Control Image Creation: Elevate your artistic control by guiding the image generation process with additional information. You can utilize sketches, line art, depth maps, or even character pose transfers to provide the AI with a stronger foundation for generating content that aligns with your vision.
• Resolution Flexibility: Work efficiently at any resolution thanks to the plugin’s automatic scaling capabilities. It adjusts AI model requirements based on your chosen resolution and scales the generated results to fit seamlessly within your image area. This eliminates the need to worry about compatibility issues or sacrificing quality due to resolution limitations.
• Upscaling Magic: Breathe new life into your existing images by leveraging the plugin’s upscaling capabilities. Enhance the detail and quality of your artwork by upscaling them to high resolutions (4k, 8k+) without compromising on sharpness or clarity. This is particularly beneficial for preparing your artwork for printing or exporting for large-scale displays.
• Job Queue and History: Manage your generation tasks efficiently with the integrated job queue. This allows you to queue multiple inpainting or generation requests and prioritize them based on your needs. Additionally, you can access a history of past generations and prompts, enabling you to easily revisit successful results or refine previous attempts.
• Presets and Customization: The plugin caters to both beginners and experienced users. While versatile default presets are available for quick and easy results, you can delve deeper and create custom presets tailored to your specific artistic preferences. This includes selecting specific Stable Diffusion checkpoints, incorporating custom LoRA models (latent diffusion representations) for stylistic control, and fine-tuning various settings like sampling algorithms and noise levels.

Conclusion

So we’re at the end of this full immersion in Krita’s Generative AI plugin… With its intuitive interface, robust functionalities, and extensive technical capabilities, Krita’s Generative AI plugin stands as a powerful ally for artists seeking to enhance their creative process and push the boundaries of digital art creation! If you’re an artist, go and download it: it’s free!

( text taken from this page from https://didyouknowbg8.wordpress.com/ )