I was fortunate to spend 15 formative years managing front office technology in Investment Banking. I was especially lucky that this covered the period when the market went under the most intense technological change. From literally hundreds of manual single stock traders in the late 90’s to more than 80% electronification less than 10 years later. The role of technology went from being a useful tool to the fundamental lifeblood of the business, both in terms of operational risk and innovation.
As we move into our third year building Velox, I find myself drawing constantly on what I learnt at the coal face about producing, maintaining and monetizing a high-quality software-based product. Here are some of the most important:
No one cares why the system failed
Success is 100% driven by what the end client experiences. Beautiful code does not matter if you don’t build in good enough redundancy and monitoring to stop problems before they become “client facing”. It is never a question of if a failure will happen but, rather, when. Major outages are usually the result of an unlikely and unfortunate timing of inter-related events Paranoia is your best friend, and as much passion and creativity needs to go into thinking about failure modes and how to detect and recover from them as system design.
You have to solve for both the tech problem and the people problem
Just having great technology is never enough. It’s built, supported and sold by people. Building a culture that attracts, develops and truly motivates talent, both as individuals and in teams, is just as important as ensuring they have the right technical skills. It’s quite easy when your company is small, but it quickly gets beyond that and then your culture will be the thing that ensures the right thing happens when you are not around.
70% of resource is spent away from coding
And 70% is if you are lucky. At a startup, managing your burn rate is so important. You have to know where resources can get wasted and plan to avoid it. From the get-go there needs to be a sound strategy around the ongoing curation of automated tests and support tools, along with good metrics to track. Testing and support can be a much lesser burden if you build things right.
The best developer is worth 10
Vitally important in an IB - and literally life and death in a startup where there isn’t the time or the money or the strength-in-depth to get it wrong – is the need to be constantly looking for the best people and being prepared to do whatever it takes to attract them. The mythical man-month is real and large teams will have a compound negative effect. Small teams with high levels of talent are critical.
If scale & resilience are not designed in at the start they’ll always be a weakness
Fixing things after version 1 is an order of magnitude more expensive. So if your product is not scalable, unreliable or too expensive to operate, you will have a mountain to climb to fix once you go live. Even with the extensive budgets and resources of IB’s, this was true - translate this to the startup world and you really have to get it right first time.
It’s really hard to use shared technology
Unless the role it plays is discrete and doesn’t change, you lose a lot of agility relying on technology you don’t control. Internally shared solutions can often be even more challenging because sharing tech is a different discipline to just using your own. A startup has to grab opportunities as they come so make sure you build (or at least have access to the source code and the ability to change it) the bits of your infrastructure that give you your edge.
Building good tech requires long term vision and commitment
Software does not do handbrake turns. A platform can be built to be very flexible but it’s always within finite limits. Sudden changes of direction outside of those limits will accelerate the build-up of technical debt and erode your profitability and agility over time. So you need to make sure you have sufficiently thought through the vision of how you will monetize your technology so you can stick to it.
The starting point is always that tech should be easier than it is
You have to accept that the expectation will be that building software should be easy (or at least easier). Because it’s possible to cut corners and deliver something in a tenth of the time - only for it to fall apart a year later when no one remembers that corner - perceptions of what is possible can get very distorted. You have to be prepared to invest time with clients up front to lay the foundations for understanding that Fred Brooks is still right; that there is no silver bullet; and why pushing too hard for an early delivery is basically mortgaging the future.
Entropy / disorder increases over time
Time’s arrow, second law of thermodynamics etc. all apply to software. All things being equal, over time entropy increases and technical debt creeps in, no matter what you do. But you can slow it down by actively managing the code base, meticulously deleting dead code, conducting proper code reviews and embedding code quality and test coverage tools into your software development lifecycle.
It doesn’t work unless you’ve tested it properly
Be this functionality, a monitoring system or recovery process. It does not work unless you tested it in an environment that is identical to the one that will be used in production. There are so many variables and moving parts such as input data, concurrent users, network and host (virtual or otherwise) that are much more likely to fail than work. Testing is a deep specialization in its own right.
In summary, building great software and successfully monetizing it is hard but by focusing on the right things, it can be done well and efficiently. The discipline of software engineering is still relatively immature (especially in capital markets) and techniques and tools are improving all the time. Similar to the benefits cloud businesses have seen by leveraging platforms like AWS, application platforms will likely drive much progress in the next decade. They will provide solutions for much of the above in specific domains, allowing a business to focus on developing the functionality that differentiates them.