Simplify a Running Django Project
Reorganization for Scalability and Collaboration
TL;DR — Simplicity is Key
The essence of the reorganization journey was all about simplicity. By decluttering the codebase, we embraced a more modular, streamlined approach, eliminating unnecessary complexities like custom preprocessor variables and scattered files. This shift not only made the project easier to manage but also set the foundation for better scalability, faster onboarding, and smoother collaboration. The real takeaway? Sometimes, less complexity means more efficiency — simpler structures lead to better outcomes in the long run.
For those who want a more detailed analysis of why, read on:
I guess every developer has faced the pain of navigating a disorganized codebase. The tangled web of files and directories, the confusion over where to place new features, and the frustration of onboarding new developers often point to a common issue: poor project structure, or in other words: a messed up project where everybody is constantly raising the question: Why the heck is this file in that folder? As your Django project grows, these challenges can escalate, leading to decreased productivity and increased technical debt. Reorganizing your project is not just a cosmetic fix — it’s a strategic move that can significantly impact your project’s scalability, maintainability, and collaboration potential.
In this post, I’ll share the journey of reorganizing our Django project, explain why it was necessary, and discuss how the new structure benefits our development process.
Django Already Bootstraps a Well-Organized Structure
Let me get one thing out of the way before I start: Django comes with everything you need to get organized right out of the box. “startapp” and “startproject” lay the groundwork, and the recommendations for structuring templates, static files, and the rest of your code speak a clear voice. However, as your project grows, it can be beneficial to add an extra layer of organization to keep things in order.
That said, let’s dive into my experience.
The Need for Reorganization
W hen we first started our Django project, it was small and manageable. But as the project grew, so did the complexity. We began to notice several issues:
- Onboarding Challenges
- Code Management Challenges
- Scalability Issues
Let’s talk about the most obvious and easier-to-discuss topic first.
Onboarding Challenges
When onboarding new developers, we faced a series of challenges. Generally speaking, bringing new developers into the project was as difficult as the code was hard to read —or flipping the direction when the code is difficult to read, onboarding becomes equally challenging.
- Challenge: The lack of a clear structure meant that new team members struggled to understand how different parts of the project fit together, slowing down the onboarding process.
This highlights a common issue in software projects. Without a clear structure, it’s difficult for new developers to grasp how different components interact, which slows down their ability to contribute effectively. Looking back, it’s easy to say but hard to tackle in “real-time”.
- Challenge: We had to integrate external pieces of code, either from templates, external repos, or re-using code from internal projects — that’s where things got really hairy.
I add this because it can be an important point when you have to incorporate external code it can add layers of complexity, especially if that code isn’t well-integrated into the project’s structure.
- Challenge: The code base continued to grow and, during iterations of different approaches to code organization, could have been better managed, with new developers getting lost in a poorly organised file structure.
This as well is easily written but incorporates a series of nightmares regarding the hard work you put in at first and then wipe out in your next “iteration” — which is the next approach to optimizing things. It highlights the cumulative effect of poor management as a project grows. Without careful organization, the file structure can become increasingly difficult to navigate, making it difficult for new developers to find their way around.
This brings me to the harder-to-tackle series of problems.
Code Management Challenges
- Challenge: Finding the right file or function became a frustrating task.
Although Visual Studio Code or other IDEs bring excellent search functionality, it’s frustrating to waste your time searching for the right code. Of course, this is a common experience when a project’s structure isn’t well-organized, making it difficult to quickly locate specific pieces of code.
- Challenge: We didn’t separate concerns.
Separation of concerns is a fundamental principle in software design. When this principle isn’t followed, it leads to tightly coupled components, making the codebase harder to manage and understand.
- Challenge: Use of custom settings for static files, and homegrown preprocessor variables for assets
This “technical challenge” is one of my favorite nightmares, because you can get so lost in the code reorganization that you have to roll back the whole work to a former commit. Custom settings for static files or preprocessor variables for assets can introduce complexity, especially if they aren’t well-documented or consistently applied across the project. If the project structure doesn’t support these customizations clearly, it can exacerbate the difficulties in managing the code. Or, break your code.
Scalability Issues
The project became increasingly unwieldy as we added more features. The various solutions we tested either couldn’t support the growing complexity or actually made the complexity worse, making it difficult to scale effectively.
To be a bit more precise, in our attempt to scale, we tested splitting apps into microservices to handle specific features independently. While this seemed promising, the complexity of managing dependencies across apps grew exponentially. We found that without a strong organizational strategy, splitting apps only made the project harder to manage. Additionally, we experimented with using CDNs (Content Delivery Networks) for static assets. This helped reduce load times and made our project more efficient, but only once we had properly integrated it back into Django’s “native static file” system, which I will come back to later. Balancing these two approaches was key to improving both scalability and performance.
The Bottom Line: The Goal is Simplicity
These challenges led us to a crucial realization: we needed to reorganize our project to improve its maintainability and make it easier for current and future developers to work with. But with a different paradigm: Reduce complexity through a simple and modular approach, not by adding layers of technical complexity.
The New Project Structure
T o address these issues, we decided to overhaul our project structure. The goal was to create a layout that was intuitive, scalable, and easy to navigate.
Get AndyEl Maghraby’s stories in your inbox
Join Medium for free to get updates from this writer.
To cut a long story short, because it’s actually quite trivial to understand, here’s a simplified “mockup” of what the root structure looks like.
(Just a “safety” note: this is all made up and simplified, of course, to make the general project idea of the organization somehow visible here; the structure is completely handmade and does not reflect our actual real-world project.)
A Simplified, re-organized tree structure of the Django project.
The picture above shows a mocked-up version of the basic structure of our re-organized code structure. Here are the key changes:
- Apps Directory: We centralized all Django apps under the apps/ directory. Each app is now self-contained, with its models, views, templates, and management commands organized within its own directory.
- Core Directory: The core/ directory now contains all the settings and configurations essential to the project’s operation. This separation ensures that core functionalities are isolated, making them easier to manage and update.
- Common Directory: A new common/ directory was introduced to hold shared resources such as models, utilities, and static files. This ensures that common resources that need to be accessed by multiple apps are centralized and easy to manage.
Of course, this has to be considered well and handled with care, because it can block portability and re-usability for future projects.
Next is a deeper view of how we organized the usage of common apps:
Last but not least, the apps folder, which is pretty straightforward and in a typical Django structure.
We left out the “templates” folder view because I think that is a separate topic and we could endlessly discuss how to structure templates into layouts, includes, pages, dashboard, sections, and so on.
Usage of native Django: In addition, we reduced or eliminated the use of custom pre-processor variables, like %ASSET_PATH%, to minimize custom-generated dependencies and increase manageability. By relying more on Django’s native features, such as the {% static %} template tag, we simplified our asset management and reduced the potential for errors in the codebase.
In other words, we actually went back to the native Django file structure after trying other scenarios of replacing static tags with the use of custom preprocessor variables/custom tags. Here’s why: When we introduced custom preprocessor variables like %ASSET_PATH%, it initially seemed like a clever solution to manage assets more flexibly. However, these custom variables quickly became a source of complexity. They often weren’t well-documented or consistently applied, leading to issues with our static file configurations. We faced problems especially when deploying across different environments, and managing these variables took up too much developer time. Ultimately, rolling back to Django’s native {% static %} tag simplified asset management and improved stability. And we are happy again with it! ;-)
Why This Structure Works:
- Separation of Concerns: By grouping related files together, each app is responsible for its own functionality. This modularity not only improves code organization but also makes the project more scalable.
- Shared Resources: The common/ directory centralizes resources that are used across different apps, reducing redundancy and ensuring consistency.
- Modular Management: Keeping management commands within the specific apps they relate to ensures that backend tasks are modular and easy to maintain. This also prevents the core management structure from becoming cluttered.
- Scalability: As the project grows, adding new features becomes more manageable. Each new feature can be developed within its own app, minimizing the risk of creating dependencies that could complicate the project.
Pros and Cons of the New Structure
Like any major change, reorganizing a project comes with its pros and cons. Here’s what we’ve learned from the process:
Pros:
- Improved Maintainability: The new structure makes it easier to find and update code. By isolating different project features into category folders, much like isolating features into separate applications, we can add new features without worrying about unintended side effects in other parts of the project.
And yes, I agree, there is the caveat of maintainability with a shared resource like “common”, but that’s a very philosophical aspect of code reusability that our answer was about: We don’t want to build a template shop, we want to make this project as efficient as possible. - Enhanced Collaboration: With a clear structure, it’s easier for multiple developers to work on the project simultaneously. Each developer can focus on their own app without stepping on each other’s toes.
- Scalability: As we continue to add features, the modular nature of the new structure makes it easier to scale the project. New apps can be added without disrupting the existing codebase.
- Streamlined Management: Keeping management commands within their respective apps helps maintain a clean and organized project structure, making it easier to manage the project’s operational aspects.
Cons:
- Fear to fail. It’s easier to keep going with a messed-up setting and build workarounds instead of wiping out the mess and starting over.
- Initial Investment: Reorganizing a project takes time and effort. We had to pause development to restructure the codebase, which required careful planning and execution.
- Migration Challenges: Moving from the old structure to the new one wasn’t without its challenges. We encountered issues with broken imports and outdated references, which needed to be fixed before we could fully transition.
Despite these challenges, the benefits of the new structure far outweigh the drawbacks. The initial investment in time and effort is paying off as the project becomes easier to manage and scale.
Future Impact on Coding and Development
Moreover, the reorganization of our project has set the stage for long-term success. With it, we can expect to influence our future development practices for streamlined development. With a clear structure in place, adding new features and integrating them into the project is more straightforward. Developers can work on individual apps without worrying about breaking other parts of the project.
As Scalability was an issue, the new structure will allow us to scale more efficiently. We can add new apps and features without messing up the existing codebase again, ensuring that the project remains stable and maintainable.
Conclusion
R eorganizing our Django project was more than just a housekeeping task — it was an investment in the future of our project. A well-structured codebase improves scalability, simplifies collaboration, and makes onboarding new developers a breeze. While the process takes time and effort, the long-term benefits are undeniable. If you’re struggling with a disorganized project, take the time to restructure. Your future self — and your team — will thank you.
This experience cost me endless hours of iteration on how to organize the project. I am sure there are better ways of doing this, and nobody will find a one-size-fits-all solution. However, it worked very well for us.
Please give me your view on my way to organizing a Django project, I would love to hear your comments and ideas, or share your experiences in the comments below, and let’s continue the conversation about the importance of project structure in software development.
