Telecom Truffle
Posts
Build it and they will come. Won't they?

Build it and they will come. Won't they?

Gergo Toth
February 07, 2024

When companies hire you for doing something you’ve done before, they expect you not to repeat past mistakes. What they don’t expect is you coming up with new and creative ones instead. Here are my ones from building a small telco, twice - by guest author Peter Orlovacz

Peter had been part of the start-up team in two MVNOs in the last 6 years, working on mostly technology and products. The experience of building these companies from the ground up gave him a unique perspective on how to build things and what’s worth building

You don’t start a new telco every day. Having worked on building two MVNOs in the last few years, here’s what I learned.

2016 - let’s go ‘digital’ - whatever that means.

The country has long had two major telcos, so the third one (well, 2.5-ish, as a B-brand; technically an MVNO without a separate license) had to be somewhat different.

The main focus was differentiation through customer experience and creative pricing. Importantly, we had a lot of freedom in figuring out what to do and how to do it. It was one of the best jobs I’ve ever had. We took a lot of inspiration from the best of the internet, especially when it comes to customer experience.

Technology was a mix of greenfield (all the frontend) and a rewrite of existing, proven middleware. There was a twist though - our vendor decided to modernize the middleware, with the buzzword of that year: microservices.

The team ended up cutting the application into 20-ish services. I lost a good amount of hair during the next year, figuring out why things were so slow and why our brand-new servers were at full capacity at a fraction of the planned customer numbers.

Here are a few things I learned:

Customers don’t always give a damn about their “experience”, especially if it only covers part of the product. In telecom, most of the product is standardized, thus hard to differentiate. Having a cool sign-up flow is great, but working mostly on that is like polishing the car dealership while wondering why the cars don’t sell that well. Because we sell the same cars as everyone else, that’s why.
Niche products work very well in niches, but remain there too. If your expectation is a large scale, maybe many of these would work, a single one won’t.
Microservices mean much more overhead just to run, especially on older frameworks. The first rule of distributed computing is still to avoid it as long as you possibly can.
It's way more difficult to debug an application that’s split into 10. We added extra observability tools and log aggregation, just to see what's happening.
Performance / latency. Sending data over a network will always be slower than just moving it in memory. A surprisingly large percentage of developers have no idea how much time basic operations (memory, disk reads, network transfer) take.
Consistency. When you split up an application and put a network in the middle, you don’t just add latency, but also the risk of failure. You can send anything through a network, but you can’t reliably make it get there. So we needed to add all sorts of extra safety mechanisms to save us … from ourselves.
Microservices is a technical solution to an organisational problem: many engineers work on a single, large system without holding each other up. This may work in theory, but the practice is much more nuanced. Be careful to check if you have this problem - we did not, we ended up with more systems than engineers, which slowed us down a lot.

We were right to evolve a previous system that worked instead of building something completely new though. The way and direction of the evolution, on the other hand, was highly suboptimal.

2020 - can you do one more like that?

After a few years, I was recruited to build something similar, this time in another country. When an engineer gets to build the same thing again, it’s usually prime time for the second-system effect to take over: Do. Everything. Properly.

This would go well with how telecoms, a world of high up-front investment, large, risk-averse and slow projects. Getting everything done once and for all, per the book is encouraged by risk-averse management and vendors (who can bill much more this way) too. Our parent operator was also a fairly old-school one, with everything technical already outsourced.

Of course, I had to do something else. (Not just because we had about 6 months to do it all)

The core telecom parts (network, charging and billing) are quite straightforward and the vendors who build these know what and how to implement. It also does not matter that much, as - having to follow standards and regulations - this does not provide much room for differentiation.

Here are a few things I managed to do differently:

it’s probably cliché by now, but the most expensive way to get software is still building it yourself. In reality, it’s the second most expensive.
The most expensive way is having a vendor build (and maintain) it for you. In the previous company, we spent months building a system for mobile push notifications - after all, we were unique, and so were our needs (not really). But this is just the beginning - it took 1-2 engineers full-time to deal with the bugs and add new features.
There were a few things we built in-house in my first company, but for the second time, I bought it off the shelf. Yes, it costs $30-40k a year, but still better than paying 2 engineers (and the vendor’s margin) for getting much less done.
Anything that’s not super-specialized - like whatever’s needed for a mobile app - is getting commoditized super fast, now you can do the same for about 30% of this price.
Invest in differentiation (=build yourself) and what makes your offers and products flexible, be frugal with things that don’t differentiate. (Like reporting, revenue assurance and the like, which you don’t have much use for in the beginning)
Premature optimisation is the root of all evil. Telcos want to build for eternity (or at least years), but that’s not practical when you are starting up and need flexibility, or when you have a small team. Go step by step.
“Enterprise-grade” software does what it says on the box (mostly. Sometimes sales teams promise things the systems cannot quite deliver), but nothing else. If you need to change anything, that will cost a lot and take even more time. (Either of which usually kills whatever business case you had.)
We ended up with a mix of open-source and in-house development. This brought a different challenge. These systems are a bit less stable, but much more flexible and you need to keep a team to develop and run them. It’s said the most expensive way to get software is to build it yourself, but after receiving support offers from enterprise vendors it turns out false.
Have a small team and don’t hire until it hurts. For example, charging and telecom operations (excluding the physical network) can be run by a handful (think 2) people.
Outsourced (offshored) development sounds like a good idea to save money, but be careful. Our team almost put us on the way to bankruptcy by choosing an enterprise access management software that costs more per user than our ARPU. (The reason? “This is what the team knows” )
When the team has no skin in the game (they will just move on to another project if this does not go well), the speed and the quality will be light years behind what your own team would do. (Which is what I did in the previous job, so I could very well compare)

All in all, I learnt a few things:

Building a ‘startup within a big company’ is mostly a mirage. You’ll end up being more like a big company than a startup, with none of the potential upside (and none of the potential downside either).
Having a mobile app is not a real differentiator, even if your competitors are from the Stone Ages. It will appeal to a few customers, but you’ll run out of growth in a month or two.
Try to do things differently in places that matter for your customers (different products, prices, comms, approaches).
Where technology cannot differentiate, keep it as simple and cheap as possible. Where it can, do it yourself.

Thank you Peter for sharing your stories. 👍️

👓️ Read More

The views and opinions expressed here are only those of the guest author.