Why software engineering is hard in a large bank
For the past 12 months, I have had the privilege of working in a large bank. And by large, I mean one with more than 200,000 employees. For those in Singapore, this is more than the total number of secondary school students, generally aged 13 to 17, and close to the total number of primary school students, generally aged 7 to 12, as of 2024. For those outside Singapore, this workforce exceeds that of Alphabet, Google’s parent company, as of 2025.
Before joining, I had never worked in such a large organisation or in the financial industry. One reason I chose to join was that my interviewers shared that the bank supported developers’ growth beyond technical skills. Furthermore, my interviewers clearly enjoyed their work and the people they worked with. Today, after a full year, I can safely say that I have experienced everything they described and more.
I have worked with talented teammates who have fundamentally changed the way I debug. While I have used breakpoints before, I have never quite used them as extensively as I did here. Being able to step through execution and inspect computed values has been invaluable when working with a mature codebase.
I also dabbled with thread dumps to analyse performance issues and honed my ability to balance competing priorities. In any given week, this might mean developing new features, reducing technical debt by improving logging or documentation, upgrading dependencies, fixing known non-critical bugs, and tackling the occasional critical production issue. At a higher level, I have listened to discussions about whether a feature should even be pursued. These conversations consider expected time and cost savings, opportunity costs, likely frequency of use, and how long the feature is expected to remain in use. Some features have a short lifespan because they are needed only during an interim migration period.
Furthermore, my team uses Scala, and I am thankful for it. Scala has deepened my appreciation for functional programming. It encourages small, composable functions, which can improve readability, and domain-specific types, which can help the compiler catch more errors. Also, functional programming encourages the use of pure, deterministic functions, making behaviour easier to reason about and test. That said, Scala is hard to master and can easily become a nightmare to read and debug. While implicits are powerful and especially beneficial for adding new features to an existing type, they are also particularly annoying to debug because they often do not appear when tracing through the code. IntelliJ can display which implicit is being applied, but it can make your codebase much harder to read.
The organisation also invests heavily in the growth of its employees, especially fresh graduates. For example, I have had frequent opportunities to interact with senior leaders. They often proactively reached out to the Early Careers Council for assistance with organising fireside chats. The council’s existence is in itself a testament to the organisation’s commitment to nurturing the growth of new employees. Apart from chats with senior leadership, the council would actively organise sessions that allow different teams to learn from one another. Together, these sessions have helped me better understand how we create value for our clients and how my work aligns with the organisation’s broader goals. Before joining, I did not know how sophisticated algorithms could be used to reduce the market impact of large transactions. I did not know that a client could use separate entities as its broker and custodian. I did not know that central counterparty clearing houses, or CCPs, use novation to reduce counterparty risk. Honestly, I did not even understand the differences between buy-side and sell-side firms, or between high-touch and low-touch trading.
Apart from honing my technical skills and domain knowledge, I have gained a better understanding of the operational complexities in building software within a large bank. I believe such complexities are similar across large organisations. More importantly, I believe understanding them is the first step towards proposing new processes that can make software engineering easier in large banks and, hopefully, other large organisations.
As such, some of these challenges are detailed below.
Complexity compounds quickly
I once worked on a ticket to migrate a small feature from a legacy system. The feature was trivial to implement. However, testing it was not. End-to-end testing required a trade to pass through at least six distinct systems:
Before
A → B → C → D → E (my system) → F (legacy system) → D → C → B → GAfter
A → B → C → D → E (my system) → D → C → B → GTo be sure, testing across six unique systems would not have been necessary if my system had produced the same message as the legacy system. However, while making the change, I discovered a bug in the existing flow. Fixing this bug meant verifying that the new message could travel successfully from my system to the final downstream system, System G. Furthermore, an existing trade must be present in System B before it can be processed. Otherwise, any message sent by my system would be ignored by System B and would not reach System G. Also, it was simpler to replay a trade from System A than from System B.
Unfortunately, the more systems a feature depends on, the more likely an unrelated change will disrupt testing. For example, systems may be temporarily unavailable for maintenance in lower-lane environments. Another developer’s test may conflict with yours. If data in lower-lane environments is not in sync with production, new errors may surface. Finally, failures in downstream systems may not propagate back to your system. A centralised logging platform usually helps us trace such failures. However, not every system is integrated with the platform. Hence, diagnosing such issues may require reaching out to the relevant teams via email or direct messaging.
Complexity reduces visibility
Relatedly, maintaining high-quality data used by hundreds, or even thousands, of internal systems is difficult, especially when there is little visibility into how each team uses it. Often, “wrong” data is corrected in downstream systems. This is because it is exceedingly difficult to identify if data is truly wrong across all downstream systems. It requires coordination across multiple teams. All affected teams must confirm that the data is incorrect and that changing it will not disrupt existing flows. Consequently, while there might be more duplicate code, it is often more practical for downstream systems to patch the data themselves. This is faster to implement and has a smaller blast radius.
Personally, this experience reminded me that technical debt can sometimes be both acceptable and necessary. As developers, our job is not to optimise for clean code, but code that best fits the ever-changing needs of the business and its users. These goals are not necessarily mutually exclusive, but we must avoid optimising for one at the expense of the other.
Complexity amplifies Murphy’s Law
Anything that can go wrong will go wrong
— Murphy’s Law
Neither systems nor people are infallible. Systems break down while people make mistakes. Hence, we must learn to trust but verify. While this might sound easy, striking the right balance is hard. Too much trust can prolong debugging if we investigate every other system before checking the one we assumed was working. On the other hand, excessive verification imposes that cost upfront.
I have learnt to manage this trade-off by steadily deepening my understanding of the systems that my team interacts with. To better understand how everything comes together, I also find it helpful to learn not only what needs to be done, but why. For example, a clearing participant on an exchange may settle several types of trades. These trades may involve an external entity, separate internal entities, or transfers between accounts within the same entity. The internal bookkeeping differs across these cases. To reduce settlement risk, both sides of a trade are matched before settlement. It is not sufficient for client A to report buying 100 shares of Apple for $3,000 from client B. Client B must also report selling 100 shares of Apple to client A for $3,000.
Get Sean’s stories in your inbox
Join Medium for free to get updates from this writer.
Understanding the purpose of matching helps me to understand why trades involving another clearing participant cannot be matched within our own systems. A participant acting for client A may receive only client A’s trade details, not client B’s. Hence, matching can only be performed at the exchange where trade details from both clients are available. Such domain knowledge has helped me to ensure that code changes, increasingly made by an AI, respect fundamental business constraints.
Regulatory requirements
The bank operates in an industry that has been shaped by numerous crises. Major financial crises have often been followed by stricter regulations and more extensive processes to reduce the risk of recurrence. Ultimately, these processes protect our clients and the wider financial system.
Some of these processes ensure that production changes are logged and communicated to key stakeholders. Others reduce the risk of developers making a harmful change to production. However, many of them introduce significant operational overhead. And when too many processes become highly manual or repetitive, there is a risk that teams focus on completing the procedure rather than engaging deeply with its intent. In turn, this may ironically dilute the careful review that these processes were designed to promote and divert developers’ time away from improving existing systems.
I was once tasked with preparing and coordinating a Change Request (CRQ). A CRQ is a formal record documenting the information required to release a change into production. Amongst other things, it typically covers the proposed change, deployment steps, success criteria, rollback procedures and key contacts.
In my experience, banks generally require CRQs before production releases. The process also requires several levels of approval. These approvals are intended to ensure that key stakeholders understand the change and can stop it when necessary. However, as some of these approvals are obtained sequentially, review windows can become compressed near the deployment date. I once moved the proposed deployment date back by two weeks to give approvers more time to review. Unfortunately, the additional time did not flow through to the final approver. I later discovered that this was because an upstream review stage was designed to begin roughly two weeks before deployment.
I also encountered a requirement to provide distinct responses in two similar fields across separate systems, even when the questions appeared to overlap. I may not yet fully understand the rationale behind that requirement. However, it reinforced how significant administrative details can accompany even a technically small production change.
Thankfully, several initiatives are already underway to automate parts of the CRQ process. I am thus eagerly looking forward to the day when developers can focus on reviewing code, and not CRQs.
Unknown unknowns
Even after weeks of preparation, several one-to-one discussions, and poring over different documentation pages, I still needed help from my teammates at 9 pm on a Friday.
A deployment command to add new alerts to a staging* environment failed because of an unexpected permissions constraint. I later discovered that these alerts had to be deployed into a separate lower-lane environment so that our application support team could focus on alerts affecting production.
Thankfully, the impact of my mistake was minor. At worst, the application support team might have received unnecessary alerts on Monday, which they could quickly disable. Still, I wondered how I could have prevented this from happening. Speaking with mentors helped me realise that I could, and should, have conducted a walkthrough of my deployment with an experienced teammate. This walkthrough would give an experienced teammate enough context to identify gaps or assumptions that I might have missed.
*This environment reads production data but writes to a separate topic, allowing new features to be tested with production data.
Low tolerance for failure
In the start-up world, founders often subscribe to the following mantra: fail fast, learn fast. The idea is that we generally learn more from our mistakes than our successes. Also, failing fast encourages rapid iteration, which can help a start-up achieve product-market fit more quickly.
In a large bank, we subscribe to a different but equally valid mantra: Slow is smooth, and smooth is fast. This is partly because the cost, financial or otherwise, of failure is exceedingly high. For example, my team processes numerous trades each day across different markets. For some markets, these trades must be successfully settled within the same day (T+0 settlement cycles). This means that even a short outage can result in missed settlement obligations, hefty financial penalties and downstream disruption for other market participants. At scale, unresolved settlement failures can destabilise financial markets, adversely affecting both large corporations and individual livelihoods.
Generally, there are two ways to mitigate this. The first way involves processes designed to prevent failures. This is more effective at reducing production issues and is arguably easier to implement. For example, an organisation might impose a temporary production freeze during a high-risk period. However, it can stifle innovation and growth.
The second approach is to minimise the costs of failures. This can make teams and systems more resilient over time. For example, a team could carry out canary deployments so that failures are isolated to a small proportion of traffic. Teams can also build robust CI/CD pipelines that allow for quick rollbacks to a stable version. However, this approach can require significant time and effort to implement.
Ideally, an organisation combines both approaches according to its risk profile and cost-benefit analysis. In a bank, the balance often leans towards preventing failures because reputational risk is real. Even if our mistakes do not result in significant financial losses today, they can erode client confidence and cost the bank tomorrow. As a senior technical leader once shared, we are in the business of trust, not software. In an era of AI-assisted development, this trust is even more valuable today.
AI adoption takes time
The past year coincided with a revolutionary shift in how developers use AI for coding. The bank has moved very quickly in adopting many of the latest AI tools. However, the frontier is still ahead. This is partly because responsible growth is important. Responsible growth requires a rigorous evaluation of any new technology. It is not wise to jump onto the bandwagon simply because we fear losing out to competitors. At the same time, this fear is not unfounded. If new tools truly unlock unparalleled productivity gains for developers, such gains will not be realised the moment an organisation decides to adopt them. The gains will only be realised after developers have had sufficient time to familiarise themselves with the new tools.
Today, AI is writing the majority of my code. However, due to various regulations, I still spend significant time debugging invalid environment configurations and cross-system integration issues. As a software engineer, I can still see the value of the judgment and context that I bring today. As someone who experiments with AI, I wonder how my role would change when AI is fully integrated with internal systems and can autonomously perform any task that a developer can. I do not think software engineers will be replaced, but I do think that the role will be transformed significantly. I will likely spend less time debugging and more time gathering context and balancing conflicting priorities so that my team of AI agents can focus on the most important and urgent tasks, on the tasks that best align with the overarching organisational goals.
In other words, I believe software engineers will increasingly need to be proficient at context engineering. Even if AI could write “perfect” code (whatever that means), it will still be hard to figure out what to write, and even harder to figure out what to remove. This demands a deep understanding of your underlying systems, your organisation’s priorities, and ongoing projects. While some of these can be found in written sources, the most valuable information often lives in people’s heads.
This does not mean that software engineers would no longer need to be technically grounded. It means that we ought to be both technically grounded and domain experts in our respective industries. As much as I unironically love the thrill of debugging (I really do), I am also excited for the future of software engineering. Engineers across all levels, but especially junior engineers, will increasingly be able to invest more time in deciding what to build, and less time correcting every last syntax error in our code.
Ultimately, software engineering is hard in a global bank because systems are complex and failures are costly. It is hard because engineers are working in an ecosystem shaped by decades of technology, regulation, and evolving client needs.
At times, this can be frustrating.
However, I believe they hone our ability to work with really talented individuals who may have different Key Performance Indicators (KPIs) but who are committed to the same organisational goals. They hone our ability to navigate complex systems, to investigate assumptions, and to evaluate trade-offs.
They hone our ability to solve hard problems. And maybe that is what matters. After all, the world will always need people who can work together to solve hard problems.
