#4 - Technical Debt

Notes

Know how to measure technical debt. It's usually an umbrella term to describe different risks and complexities in the software.

Instead of re-writing a legacy system consider evaluating its current state and looking at the product and business goals. Interviewing the people who work on the system can give you a good sense of the urgency. Look how many escalations or production issues the product has over the past months or years. Use retrospectives with your team to understand what can we do to make us more productive, to increase the velocity.

You can divide the tech debt into engineering and business debt. The latter is measuring the time to market, end-user experience, and stability. Software is enabling the business and that's why it needs to be ahead to provide the right opportunity for the business.

Identify and prioritize the work. In order to get buy-in from the stakeholders, you need to back it up with the data, estimates, and return on investment.

Big migration and refactoring are not always the right solution and even can introduce more problems. Because the complexity in the legacy system is hidden until you open it up during the migration. This can lead to the increased scope and more pressure to deliver.

Transcript

John 0:13

Welcome to pragmatically podcast, your hosts are Alex and John Masse. We have conversations with folks throughout the tech industry to get real world perspective on how people make things happen for their careers and businesses, check out PragmaticLead.com for more content just like this.

All right, Alex. Awesome. Good morning.

Alex 0:39

Hey, John, what's going on?

John 0:41

Hanging out. All right. So today, we want to talk about the recent article that you published on dealing with technical debt. We've we've talked about this a little bit back and forth. While you were working on this piece on we thought to spend a little bit of time to kind of deep dive into it and have a conversation around technical debt.

Alex 1:06

Yeah. So it's a painful topic for everyone. And anyone who works with software, at some point in their career, they will be working in struggling with technical debt. What is it, how to deal with it, so it's kind of interesting topic that people experience all the time. So yeah, it's necessary to talk about it time to time for sure.

John 1:33

So technical debt, Alex is something I thought about recently, is kind of an invisible debt. So I'm a programmer, I'm working with a designer, a product person, we have zero software right now. And now we're gonna start making code. I think I'm making a product. I think I'm doing a good job. My program is working. But at some point, the debt starts to accrue. But we don't see it. It's not like you're using a credit card, and then you can look at your statement at the end of the month, and know how much debt you've created for yourself.

Alex 2:13

I mean, the the the first challenge with technical debt, just talking about technical debt is how do you measure technical debt, it's different for every product, every company, every team. How they go about measuring in kind of thinking, reasoning about technical debt, it's different for everyone. But it's important, like, with everything that you try to improve, you start with the baseline, where you are right now. So you can measure your results, are you actually making any difference? So yeah, measuring technical debt is one of the challenges. And that's pretty much where you have to start with.

John 2:51

So you know, you and I, we've been building stuff for a little while now. My, attitude towards software has changed a lot. Since my early days of my career. And I feel like, I want to avoid creating things as much as possible. For the sake of keeping the amount of things I need to be concerned about in the future down. And that's something I think about is that like,

Alex 3:23

So with that, so as soon as you start building anything, you pretty much go into technical debt. So you start with this. It always feels really cozy and warm. Like when you start building something new, like it's a new clean slate, you start with a blank screen, in your in your editor, and you start just adding stuff. And as this as soon as you start adding stuff and you shipping stuff you're going to into technical debt, because as soon as you ship something, it becomes outdated. Now, you're gonna start adding new features, business requirements, updating stuff, libraries become outdated, security issues arise. So you started getting into technical debt pretty much as soon as you ship something into production.

John 4:16

So I definitely agree with that. So you talked about measurement. Now. There's different types of debt. You have the actual existence of code. That's one thing that needs to be maintained. And then there are the things that you didn't predict, like a bug, either a business issue, or it could be a security issue or security concern. How have you, like how would you categorize that differently?

Alex 4:49

Yeah, I mean, it's a it's an umbrella terminology, like technical debt. What is it? It's a it's a risk, could be like risk, complexity, degrading of quality of your future software or the system, it's pretty much when you borrowing something like from the future productivity. So you taking shortcuts to gain some productivity you're getting to immediately. So this is kind of technical debt. It's just an umbrella terminology. And you can you can bucket anything under this umbrella, you can put bugs, complexity, risk, you can put external dependencies unknowns, pretty much anything under here.

John 5:35

We're talking about technical debt, but I think it'd be helpful to help like to talk also about the effect it has on a business. And I like the in the post, in the post you on pragmaticlead. There's this nice chart that I think illustrates it well, where complexity is on the Y axes. And then Time goes along x. And there are these moments where, as time goes on or more, it takes more time to complete work as the complexity also increases.

Alex 6:14

Yeah, the, the more you, you build the software, or the longer your project lives, the more technical debt and complexity you accumulate, the more people you add to the team, the less context people have on the team, like what's going on, it's just hard to wrap your head around the big system, the longer it exists, the bigger it becomes, people move in and out for like from the team or the product, the company, new people come in with less context. So they start kind of adding something new. So the other team members lose that context. So that complexity builds and builds and builds. And over time, you just accumulate more and more technical debt. And this is what this diagram is trying to how.

John 7:01

So curious. And, and you and I have seen this happen a lot. There's a tendency as a programmer, or leader coming into a new team, and you're greeted with the applications in the software that were built by the people that came before you. Is it kind of a knee jerk reaction? Or is it fair to say you think that if I'm a new leader coming into a team, what it feels like, to me is a whole lot of new debt that I have to contend with.

Alex 7:31

Always, when you start a new, a new team, or a new product, always rebuild it from scratch. Just kidding. Well, that's the reaction or reflex we have. It's just like, Oh, I am faced with something new. It's just easier to build it from scratch with something that I understand and know really well. So if I join a team that I like that build something with, I don't know, like Java, Java Spring Boot, and I've never worked with this technology before. It's like, oh, why don't we build it with Python or Node JS? Because I understand it so well. And I know it can be more efficient. So I just need maybe six or 12 months to rebuild it. And that's, that's what we like to do all the time. But is it the right choice? I don't know.

John 8:19

And I don't know. That's the question, right is when do you When do you make that determination? Because rewrites there are some companies that they have a policy where they might release a version of an application and manage it, or maintain it or have certain SLA's over the course of maybe three years. And after five years of running that product, they understand that needing to basically burn it down and build something new to replace it.

Alex 8:49

Right. So I want to go back to, to to think about this, to your question, how do we measure technical debt? Because it all starts here, right? So you join a new team, and you're faced with this technical debt, and you overwhelmed and your team is saying like, oh, which is this is like emergency, we have to address it as soon as possible, or it will just collapse under its own weight. And we have to do something immediately. So how do you actually think about this? How do you make the right decision? And, of course, you want to trust your team. But there's also pressure and pushback from product and business. Like we just cannot afford to invest six, eight months into just paying the technical debt and just working on technical issues. So how do you go about this? How do you measure technical debt?

John 9:39

It's so so what I do is for measuring, like, if you're asking me that question, um, there are a couple of things that I pay close attention to, for instance, when we're going into work in a place, what's the general motivation of the engineers, okay, so sometimes you can hear grind Someones that was actually a point that Mike Gertrudes had made in the first episode, where from a product owners perspective if they want to get a sense of where technical debt might reside, is to get the boots on the ground perspective by talking to the team. Hey, guys, like, I'm thinking about doing this thing, and or this is a feature we're going to execute on. How do you feel about that? And if general concerns start to emerge, then you kind of get a sense of what you're in for. And that's at least one way to get a sense of whether or not there's debt around that should be looked at, or evaluated. Because it's not easy, especially if you're, like you or I, we've inherited applications, a lot of times we're dealing with existing art. And sometimes they're old, I mean, I've been on systems that are 20 years old. And so to understand its capabilities fully t's largely impossible, especially if the legacy or the staff or the engineers that had originally invented this product, or application have long left. And now there are these very dark parts of the system that you'll probably never see. And if you do for some reason, or if there's a problem with it, it's very hard to recover from those scenarios, at least without panicking at least. Um, so that's that, for me is one is to, if I'm not working on it myself, because that's, that, for me is the most obvious, if I'm perusing componentry, if I'm perusing patterns I can, for an upcoming code review, I can, I'm looking for signs that there are detractors to common patterns between folks. And those could inflate the amount of debt that you're accumulating like there could be two or three programmers on the team. If your PRs are just being rubber-stamped, that's, that could be a sign that you have tech debt that's just accumulating. But you can understand those from a leader standpoint if you know what decisions are being made, because then you can backfill training, talk to the team about it, invite refactoring, and I'm jumping ahead a bit into ways to mitigate or manage tech debt long term. And those are some of the things I look for also. The other one is escalations, I think how many escalations or bugs are introduced in production. And then what I'll try to do is bucket those things. So for instance, if it's for e commerce, you might have a price, a pricing inconsistency issue. And I try to pay attention to how often these issues emerge, like common issues. And those are the areas that I'll start digging just a little bit into, say, like, Well, what does this look like? What's in here? How are these decisions made? And then that starts to unravel on unveil itself as a tech debt that's been either sitting around, it's been cruft, the people have been kind of patching on top of it, to hide it away, or to try to quickly get something into production. We call it hacks around the core issue, or if we're not addressing the core issue. Those are signs to me as to where tech that is now that's now how do you measure that? What's the measurement, I guess, that you want to understand? So going back to complexity and time, if the goal is to manage the complexity for the sake of keeping people productive, then we can look at PRs. We can look at commits, we can look at the frequency of contribution or advancement in certain corners of an application or product. Those could be ways to measure that. So for instance, if you have a landing page, or an e-commerce product, and you have a list of items on it, or you have a checkout product. How long does it take your team to get a feature into production? What does that look like? What is your technical velocity? And if there are reasons or in a retrospective, if you're an agile team, if during retro you're saying, Hey, guys, how could you have done this better? How do we get our velocity up? And again, going back going back to the engineers, they're saying, Well, if we're able to clean up these few things, then we think we'd be able to move much faster. To me sounds like the beginnings of something. Actually measurable.

Alex 14:59

I completely agree. I like to think of this of technical depth is gonna break it down into two major categories. One is on a technical side, more on a craftsmanship and complexity with with the code and architecture itself, like, looking from the engineering perspective, like what's the technical debt? And what effect it has on engineers and the code base? And then the business side. What's the velocity? Can we estimate things properly? Can we deploy or ship our features to our users fast enough? These are gonna two major categories of technical debt. And when I joined a new team, or even team that I've been around for a long time, it's always you want to have this kind of 360 degree view of technical debt. Now, just think from like, from the engineering perspective. We want to just work with the cool latest tech and just refactor everything constantly, just to catch up to the latest trends. That's great. It has to exist, right? You want to keep engineers motivated and excited about things. Nobody wants to work with the old technology, even though it's fine. But not everybody wants to work with that. I think we had a chat in the past about Angular versus React. So maybe we can we can talk about that.But then the other category is business, right? So you want to talk to engineers, you definitely want to get their perspective, you want to get them the seat at the table, you want to make sure that their voices heard for sure. And then there's a business side, you want to understand, like, what's the end goal with the software, what the system. Usually is to generate some sort of business or to provide some sort of value to the end-user. So you want to understand what are the challenges? Are we moving fast enough? Of course, it's probably not going to be 100% satisfaction from the business and product people want to do is move faster and ship faster for sure. But are there any opportunities? can we do something to f get ahead of product and business? So we enable the business even more. So and bridge those two sides together and think about technical debt from that point of view.

John 17:29

I like the way you bucket it, I think that's clearer than the way I described it. But you're right, you have to segment those things. So I'm looking at your post, and you mentioned making, making paying tech debt. And I'm, and I can't wait to talk to you about frameworks. But so paying for tech debt, paying tech debt to me sounds like hey, I've had an opportunity to refactor something or I replace something, I dropped to the pendency on a library or something I was using in my application, these things are going away, and they no longer need to be a concern of mine. And once they're kind of gone or in pattern, they become simpler and easier to maintain. At the end of the day, right? We're just getting time back. We have fewer things we have to mentally or physically contend with. Is that fair?

Alex 18:24

It's one of the benefits.

John 18:27

So now I'm looking at a process. And once you have a sense of how to understand your technical debt, how do you get it into your daily hygiene of how you do work? I see here you you mentioned, using 10 to 20% is the most common ratio of feature to tech debt work. But in practice, it's never that clean, at least in my experience. I've never seen that to be that clean it in order to do that I it requires the active engagement of most of the stakeholders, you the development staff, the management in the engineers on the team they need to be they need to understand their contribution of communicating tech debt. Because if they're not talking about it, what are they spending 10 to 20% of their time on. So what do you think about that?

Alex 19:25

Well, first, I think it's unacceptable. It has to be clean. Not good enough. Of course, it's never gonna be clean. I I've seen recently I've seen the the analogy of like, I think technical debt as Tetris game. You play Tetris. There's going to be gaps, but it's fine. You still can win the game or continue to keep playing.

John 19:51

Is it a mode or B mode?

Alex 19:54

I don't know what that is.

John 19:56

You can't use a Tetris analogy without knowing the different modes. You play forever until you fail.

Alex 20:04

Let's play Tetris now and find out. It's like things go for like Tetris game, right? So there's going to be gaps. And for the most part, it's fine. You still can keep playing with the some gaps in it. But those gaps become too big. Then Game Over, you have to start again. So it's never gonna be perfect. It's never gonna between. Yeah, so let's talk about solutions. So now we know how to measure technical debt. So you need to think about business technology have this 360 overview of like, what are you actually trying to accomplish with your with your code, what your system, what opportunities you're trying to, to get, or create for the business and product and what opportunities you're trying to create for engineering. It's not really measurable like, to the point like where you can assign story points and numbers to technical debt. But that would be ideal. And we're going to get to solutions later. But that would be ideal when you try to get the technical debt prioritized, and sell it to not sell it. But to get some buy-in from the product and business. And even engineering, you want to have some numbers, something to back it up. So that's one. So once you have that you understand like, what your priorities are and what you're trying to to address. Next is you actually need to figure out how to get it done. So the process. I talked about Sprint's. 20-30% percent of every sprint, yeah, 10 20% of every sprint, something like that, right? Yeah. So I've seen it, it's pretty common, like, the team will put 10 20% 30%, whatever it is, of every sprint, they will have like technical stories, technical tasks, to constantly work on, we're going to update this library, we're going to update this framework, we're going to refactor this part of the code, we're going to write this amount of unit tests because we skipped over because we had some deadline in the past. So constantly trying to catch up, because you're never going to be 100%, clean without technical depth. So it's going to be baked into your process. And it's agreed upon with your product and business beforehand, like 20% of our resources are going to be allocated to work on technical work, because otherwise, we're just going to be in trouble in the future if we don't do that.

John 22:39

See? I think it's easier to say, than, and I think it's harder to do. And here's, here's why. In order to do that, well, you need, in my opinion, you need a distributed perspective of what debt is valuable to work on from the team, and possibly the input of a manager or tech lead, in order to actually set a priority. So if you're, you are chewing on 20% of the time to spend on tech debt, what's the most valuable stuff to chew on, and that and if you ask a team broadly, it might differ, and there might be no opinion at all, especially for if you're a new programmer coming in, all you're trying to do is close tickets and participate in being productive. But you don't really have any ideas of you know, if I was going to ask that person, hey, if I get you know, for 20% of the next sprint, what do you want to work on to pay off tech debt? They're gonna have a hard time giving me an answer for that. And so while I think that's a good number, the other thing I'm going to throw out there too, to think about is we're chopping off or drawing a line between the business, product and technology organization, or saying that look, product team, you don't have to be concerned with this, just give us a black box, and we'll call it 20%. And trust us, we're doing stuff that's meaningful. So what do you have? What can you say about that? Like, I have an idea? I, I would, I would like to push it actually a little further. I think that a healthy culture around tech that is inclusive of the business in the product, and it actually invites the question. So something actually Frank said, um, and said that

Alex 24:43

Episode Two,

John 24:44

yeah. where he's, um, he described, he invited the product team to weigh in on what they thought, a good build time was. And as tight as a technical organization, a lot of times I see us hiding the pain away from business and product instead of inviting them in to help us manage it effectively. What do you think of that?

Alex 25:11

Yeah, that's kind of what I was saying with the business side of technical debt, you want to understand what they actually trying to accomplish what we are trying to accomplish, but they specifically from the, from the business and product side, what, what velocity they want to have, how fast they want to get to market, what opportunities there are that we are not taking advantage because we have this limitation on a technical side. So you want to understand that you want to work with, with your stakeholders to understand, what are we actually trying to accomplish with the software?

John 25:42

Do you have an example where you were successful in making that connection between tech debt and business objectives? That you thought you actually made that connection?

Alex 25:58

I have many of them. So let's see, let's say migration, like the I think we talked about transformation and migrations, and then big refactors, with Frank on episode two. So let's use that example. So when you're migrating something, and I've had this big migration project from Angular to react in, I think, like, two years ago, and actually product was the biggest advocate for that project. Like product managers were the biggest advocate, they went to CTOs and CEOs, and advocated on our behalf on a technical and engineering behalf to actually get this project funded and allocate some time and resources to work on it. Because we both found so much value in this migration that they thought it would probably benefit the product the most. And some of the benefits were the responsive design. So taking advantage of components reusability, velocity, the performance of the web application. So we could do some interesting stuff with Webpack, lazy loading, and code splitting, and things like that progressive web application. So there's like so much we could do, and thankfully product understood all those benefits with engineering. So it wasn't just pushed from the engineering side. So that was a really good example of collaboration and with the product to take advantage of software to to ship the software, and features faster. So that was what was a great example.

John 27:37

So Well, congratulations. And I felt that product wind in the technical sales before. And it's your right, there's once you've made that connection. Follow up question, because I think it's it's fascinating. Once you've, once you've made or established an initial connection with your product stakeholders, do you find that they more they more or less are looking for more work like that later on?

Alex 28:09

Do you mean product?

John 28:10

Yeah. Like, are they do you think that it gives them or gives more people? And I know, we're using the term product, because we work with, we work with product people a lot. But in my opinion, I think once I find a win, there's probably more wins to be had that I just don't understand yet. And so I might, I might ask more questions.

Alex 28:29

Sure. I mean, there's always going to be more opportunities. But you need to find that balance, like the bounce is always a key, right? You don't want to like chase all that latest technology all the time, you want to like do incrementally, you want to build something and get the value. So you want to reassure that there's actual value and you're getting something back from your investment, you invested something or resources and time into that latest technology into migration into refactoring. And you want to get the value out of it, you want to prove that it's actually faster, the performance is actually faster, you want to prove that the velocity of the engineering team is actually better, you can move faster, you can actually reuse those components, can you actually prove that when the work is done? Are you actually successful stop, then stop and look back, do a retrospective. And look ahead again, like look 6-12 months ahead and think what the market and user base will be like. What opportunities we'll have in the future that we can start building towards now. So it's not always just catching up with the latest technology. It's actually thinking about the product, thinking about the users thinking about the market.

John 29:44

When you get a new application. How do you avoid having to like that knee jerk reaction to rebuild all of the things?

Alex 29:52

I guess my experience is that rebuilding things and I've done I made those mistakes. Rebuilding things is not always gonna solve all your problems, in fact, it's going to introduce even more problems, because you're going to start uncovering so much complexity in the old system in the legacy system, that you're going to be so surprised that the new Greenfield project will not be enjoyable as much. Because when you commit to that, refactoring, even when you commit to that migration, transformation. There will be pressure, there will be a lot of eyes on you, there will be some expectations to deliver fast and actually deliver on your promises. And you never know, you never know 100% that those promises will be true when you delivered a new software. That's why it's a big risk to do that.

John 30:40

Yeah, I don't know how many times it was all the new, the new thing is done, let's turn it on. And all of a sudden, the performance isn't so good. And I don't mean performance from an application runtime perspective. I mean, it from like a business perspective, it's not performing what's going on. Tt times a long tail discovery process of features and patches, or these little corner features that you didn't, you never saw before, pop up and exist that now you have to contend with. And now you have sometimes an unpredictable timeline of additional investment. Now your stakeholders are like, What gives? I thought this was only going to be a two-month thing or a three-month thing. But now it's six months? And now you're telling me you don't know? Or why is it not performing as well, you told me it was going to perform better? So yeah, I mean, after going through, I think sometimes you have to face that a couple of times before you realize it's, you know what maybe it's not actually worth it, because we can't get our arms around it. But that actually puts me into the mode back to the conversation we're having a couple of days ago about, um, you're giving me some feedback on AngularJS. And for the record, it's not my favorite thing. It was fluent calm, where it was what backbone was all part, like super popular. And Angular was, was the bigger option. And that that fluent calm that year, they're like AngularJS is the way and so we all went that way. And what was it like maybe three months later, when we learned that none of the things that we've been doing are going to stay the same? So you better get ready to eject. Like, oh, why I'm so that's kind of a that's almost like you said this in your posts when you're you're going out and open source, you're taking a risk, you're moving a bit faster, you're taking advantage of the work of others. But at the same time you are adding the risk of the future of that work, you're signing up for that either, or you're a company that forks the thing or makes a copy of the dependency and use on their own, which is different, you're taking ownership of the future of that thing. However, there's a lot of software still built in Angular.js, that solves good problems. I've worked on some of these systems. And the conversation we're having was, you know. When do you kind of dropship and make that decision to that's that large, where you're going to completely, you're going to literally build a whole new thing, or you're going to try to smush the new thing into the old thing a little bit. Maybe you're going to use the strangler pattern, maybe you'll build facades, but largely you have to make this decision to kind of cutover. My argument is if you're making your goals and you can make the current system tolerable, and people productive. Why then would I make the change, why would I then ask my CTO for the time or my leader or stakeholders for the time to build rebuild it all in and react again. If I'm already making my immediate KPIs are objectives, even though as a technologist, I know it could be even better.

Alex 34:18

Yeah, so to answer your question, I want to go back to add one more thing to measuring technical debt. One major red flag for me when I join your team or just working on any product is that. When the team cannot estimate accurately, or at least at least approximately how long things are gonna take. And the team constantly misses deadlines, because complexity is so big that they cannot actually say beforehand, how long things going to take. It creates a lot of uncertainty for products and businesses like we don't know How much investment we need to deliver any of the features. So it becomes a big, big unknown, and it's bad. And I feel like that's what was happening with Angular. And that was one of the reasons why we wanted to migrate to react. It's one of the reasons because the code became so complex, complicated, some half or more of a team member left or switched to another team and other products. So we were losing that context, all the time. So it was getting to the stage where it was pretty much unpredictable. We had production issues, lack of unit tests. And we were facing like you said, the choice to rewrite everything in either Angular two, because it was not backward compatible, or react. So rewrite would have to happen no matter what. So at that point, you have to make a decision, what are you going to rewrite in? Are you going to rewrite in Angular two, or react in that case, but in general, like Angular react aside, is that it's probably not. And this is what from my personal experience, it's probably not a great idea to rewrite everything from scratch like just take the whole thing and rewrite it from scratch. It's probably a bad idea. Now I like it to do it in stages, or in components. And like I know, microservices is a hot topic. It's, it's debatable whether it's good or not. But at least microservices like if you think microservices, you can decouple your system and start kind of rebuilding components. So the authentication, rebuild it, and ship it right, then connected through like some sort of service mesh, or HTTP or something like that, something along those lines. So it's probably a more sustainable and predictable model like just decouple your big system, and do it in phases and stages and in components. So it's more predictable and gives you that immediate validation or feedback loop.

John 37:15

But that's not always easy, right? I think the, um, so the Angular JS story you have, it seems like you have to be a little more aggressive sometimes. We pushed this, okay, we'll start building, we'll start building components, React components, and rendering them in our Angular JS application. But it gets worse sometimes before it gets better. And then sometimes you have to, you have to make a decision as to when you're going to cut over to the new thing, especially when you're mashing things together in that way. You have some code componentry that can live long term, but then it's all mixed up in this thing. That's just not, it's not good. And, and I mean, good from the perspective of, you're actively adding debt, at the same time as trying to migrate, for instance, rendering react components in Angular JS. That's a, it's doable. But I mean, I've seen I'm just all these either deferments to render, what's react responsible for what's anger versus what's Angular responsible for, the design principles were largely different. For instance, Angular JS, largely dependent on references, and then he had this dirty digest, while react has this one-directional thing. And, and so activities in one tool would invoke things in another. So in that example, I mean, while it was possible to do things incrementally, it was at the same time making the system you relied on worse on its own, and even more difficult to maintain for the benefit of starting to build these components that you hoped at some point eat away at the system that was surrounding them. Yeah, so I would never, I would invite you, I would invite you to add option three, right? option one is a redo all of the things and commits to it. Option two incrementally tries to sort it out. option three, strategically evaluate the shape of the application, and to establish standards and patterns that would help you work or start to get time back or predictability back into the current system. And that was the thing that we were talking about before was I felt like I achieved that in an older framework, where I wasn't so much encouraged by like I would probably slice up a migration like that much differently. Now I would look for larger pieces that I can replace, not the whole thing. The bigger things that I can say this is a unit. This is a closed scope. And it's not going to blend or mixed with these other things. And I can keep that safe and pristine. And if people are working in there, they don't also have to contend with all of the other complexity. But you could also steal patterns from modern tools and solutions, and see how they fit into your system or use those things to kind of change how you work day to day or draft tickets or refactoring strategies to get you into a place where there are patterning and leverage for a programmer to better understand what it is that goes into working on a legacy application or an application while legacy and I think it's the money-making software.

Alex 41:04

And Episode Three, we talked about this a bit. So don't call your legacy code, because that's the code that pays your bills. And I said it in my post, most of the world's still run some Java eight, there was a really popular blog post by forgot the person's name, but I linked it into my blog post, that while we all like to talk about go Lang and rust and all these latest frameworks and languages, most of the world's most of the software, it still runs on Java eight. It's old, boring technology, but it gets the job done.

John 41:41

That's right. Well, I love that too a lot. So that's kind of interesting. A lot of companies nowadays I hear and even managers I'm talking to from various businesses, they're kind of on this mad dash to change over to Go. Like, what do you think of that is that it sounds ambitious. It sounds exciting as a technologist. But it also sounds really expensive. And not only from the, like the operational point of view of I mean, actually, operationally, it's more interesting than some of the other stuff, maybe even cheaper. But that detail aside, there aren't a lot of go programmers out there. And these companies are actually investing in training people to write go. How the heck do you think that was sold to a stakeholder that we're going to build a university around the language to support it?

Alex 42:33

Well, it's hype. A lot of people like to jump on the bandwagon and kind of write and then because it's exciting, it's new. Like everybody likes Tesla and excited about Tesla. I don't like Tesla. I mean, I like how it drives. It's a great, awesome car, but it just I don't know... yeah, it just doesn't feel like a car that's worth $80,000. Yeah, it's just not it's I don't think it's worth it. There's a lot of negatives about the car. And I tested it. I rented the car for my road trip, and I didn't like it. So yeah, maybe hype. It's not worth your investment. Not always. So maybe that GoLang or rust is that Tesla that everybody talks about and excited about? But when you actually try it, when you migrate to it, maybe it's not as great as they said it would?

John 43:30

I suspect there must be a really big like, so if I was going to measure that, and if I was going to sell you on Hey, Alex, I want to rebuild all of our products using go or rust? Actually,

Alex 43:41

I would, I'd be excited as hell like, I'd be Yeah, let's do it. Yeah

John 43:47

But like, right, we pay our own bills. So you know.

Alex 43:52

Yeah. Yeah. I mean, there's probably a reason why companies and businesses do that. There's a lot of hype, and one of the reasons why we like to keep the technologies up to date, and upgrade and use the latest and greatest and use Kubernetes and invest in the cloud and service mash and all this kind of complexity. I think it's a talent, attraction, right? People want to work with the latest technology. So if you had a, let's say, as an engineer, John, you interview in two companies, right? Company A and Company B, Company A gives you the offer, and they use React, they use Node.js, they use graph QL. They use GoLang. They use rust, they are on Azure, or Google Cloud or AWS. They used all the latest Kubernetes it's all kind of figured out they have great people who understand that they can teach you that you can get that experience and Company B gives you exactly the same offer. But they use, they use Angular, they use Java eight, they are on-prem you have to kind of manage your own servers. It's a bit data, it's a bit old, it's a bit more conservative. But you know, the offer is the same. So as an engineer with which company you would choose.

John 45:10

I mean, for me, it's obvious, I would go with the folks that are more on the edge, they're more up to date. And they're not lagging behind. There's and the reason why is there's that, to me, there's, there's a connection between the business and the technology if the technology is lagging really far behind. That means that the business and product objectives are overwhelming the capabilities of what technology might be able to accomplish. And so there, there's, to me, that's a sign that there's a culture rift in there. And there are conversations that have not yet been had. And for that reason, right, because I would ask the question, I would, I might, I would even go back to the legacy folks and or the folks that are having a hard time keeping their tech up to date, or paying tech debt. I think it's more appropriate. And say, well, what's the initial question that I, I like to ask folks, what's your appetite for change? How do you communicate business value, how engaged are these teams because I am also not the kind of person that would avoid a challenge. So I don't need to I'm at the point now in my career, where I feel like if I need to write a react thing, I can do that if I need to stand up a full-stack thing, I can do that. But I can also help a company move from a legacy state to a modern state. For most of my career, I've been doing about doing that. So I wouldn't back down if I felt that there was support within the company. So then at that point, I'm looking at, well, what's the brand, how's the product positioned? Which product Am I more interested in, because I should also, I think, play apart, for instance, if you're going to, you know, do something that's gonna, your job is going to push you way back in the back corner, you're not really going to have an impact, or, you know, the other company is going to give you a broader responsibility where you have to gradually provide education help, people, along, and that is much more interesting, cuz that's going to challenge me a lot more than just going in and kind of digging into the easy stuff that I know is gonna work that I know is going to be easier to grok because there's been a time that's been put in to make it simpler and more approachable. So I just want to play both ways. I think it's easy to pick the place that's up to date. But also, if the legacy company was understood that they have the real change they need to make for the benefit of advancement. That's also I think, very admirable and worth considering.

Alex 47:52

Right. So usually the company A is never the the the modern company is never that modern, it usually the story, they tell you, we are going into the cloud, we are migrating to Kubernetes. So it's a, like you said it's a sign that the company is willing to take a risk and willing to make an investment in technology. And the company B is not willing to take risks. I think you said that before. Like if Angular works, why would you need to migrate to react? same kind of mindset, right? If it's working, why would you change anything about this? Well, yeah, and that's kind of like from the candidate point of view is that that's concerning. Because as an engineer, it's kind of like why people get into engineering software engineering in the first place?

John 48:42

Well, well, wait a minute, because, okay, because now you're saying that the decision to not change something because we can maintain it? So it sounds like what you're saying is, you're looking at the decisions. And if we can actually manage our debt or manage an application that's probably using a language or framework that's a bit older, but it's not causing any pain, you would still count that as disengagement from advancement.

Alex 49:05

Well, yes, and no, the answer is usually more complicated than just yes or no. Would you give you an example? Would you go and work with a company that uses what's the pre jQuery libraries?

John 49:20

pre jQuery?

Alex 49:21

Yeah, something or even like j early jQuery, jQuery, mobile, jQuery UI and things like that? Sure. Of course, it's gonna work. Of course, you still can build the UI on top of it. But should you? What's the earlier version of Java or even like PHP, not the latest PHP, but the early versions? Like it still works? Should you or should you upgrade to the latest version of Java, not the latest but maybe like Java eight or Java 11, where you can take advantage of a lot of the new stuff,

John 49:56

Well, assuming that application those features are running into the, to the goals of the application that's being built?

Alex 50:04

Well, it's relevant. And also, can you actually innovate on top of what the language provides you? Like, can you take advantage of the lambda functions in Java? Can you do some interesting stuff with functional programming without, you know, just going old school and building everything, you know, as a classes object, object-oriented, I'm not saying there's anything wrong with that, I'm just saying there are probably newer, more efficient, better ways to do things? And you should kind of at least, approximately trying to catch up with what the latest tech, and you better off than stuck with whatever version you do start it and just stuck there.

John 50:45

well, let me give you like, I'm gonna I don't, I'm not hearing it quite yet. Because I understand what you mean, I think it's really easy to pick on the decisions of the past, as, Oh, you're kind of stuck there. Um, but unless you have real pain, and the programmers are going like this is just not working and to the chart, if the time becomes unpredictable. But if you're still in a very predictable state, I would argue that that's, that might not be the right way, especially if you're a small tech team, and you have a lot of different types of things you're trying to do or promote or put out there. It's probably better to spend that time elsewhere. Now, unless, for instance, you have something that just isn't maintained any longer. And AngularJS is a good example. And you know, the further you go away, you detract from that, the longer you wait, the less support you're going to have. And that's a real business per business reason, I think, is that the number of people actively working on that technology, that pool is actively diminishing, and you need to think about what your position is going to be in the next three years. So that you can, and I know it's a relevant thing. But you won't, you want to be in a position where you have the tools in place, not only that people are will engage in but that are actually relevant and relevant from the perspective of that there's a community behind it that's still working on it, that the documentation is still improving. And that those are the things that I think also are ways to talk about the value proposition of doing something that heavy, but at the same point in time, you know, if you have all right, we are we have some software that's using some legacy stuff, like handlebars, right. And those things still work. And that software still works. But if we were going to rebuild that, like, do we spend the time now to rebuild all of those features, with react and progressive web application stuff? Or? Or do we build the stuff that closes the feature gaps to make the application relevant?

Alex 53:11

I mean, I agree. It's not always about what the latest technology use. And if, if it gets the job done, maybe you shouldn't, all I'm saying is that part of the reasons why companies try to upgrade to the latest technology is not only to meet the business requirements, and create opportunities for the business and product, but also to attract people. If a company has to hire a lot of people, they will have a bigger pool of people, like you said, the pool of people that are familiar with that technology that they can hire from, if they use something old and outdated, not maintained, it's just going to be much, much more difficult to find people. Yeah, that's a challenge.

John 54:00

So now looking back at go and rust now. They're not known as the most popular languages currently. And any company at least today that's interested in adopting these technologies are also buying the education of the staff. They're engaging in and working with it. And what's the value proposition?

Alex 54:26

Well, I cannot answer that question. Because I haven't had experience with rust and go and specifically these languages. So I cannot say why but I would say it's part of it is the hype and people a lot of people excited about this. And a lot of people when they see rust or go on a job req, they're more probably more willing to join that company because it shows the company mindset and culture is to be more modern, to invest and take risks in technology.

John 54:55

I like that. So there's I don't want to call it the sales pitch, but you want to have at least a little bit of risk with, or maybe it's not risk, but you want to be a little more liberal, in some areas, not too liberal, but a little bit, just enough to tell the world that we are experimenting with these technologies, we're looking at them, we're considering them seriously. And that very well could be a part of your career or exposure while you're working with us. But at the same time, I'm like, how do you make these really big? Right? When it's very big, and it's very aggressive? I have a hard time swallowing that pill, when we're gonna we're gonna completely change the language that we're writing with. I mean, look at how expensive it was to get out of if you're writing JavaScript. Version five, ES5 version five versus version six. There are still people that struggle with understanding the differences between some keywords, some of the keywords in the language. And that was expensive. So now if we're going to take those same people and say, Okay, we're gonna take, we're actually gonna give you a completely different language to create stuff with, I have a really hard time with, at least without strategically applying it, as I look at rust, I look at languages and runtimes, like rust as I'm beneficial for the cloud era, Reason being is the footprint is much smaller, and you can get a lot more process power out of that application out of that runtime. And I would say go is absolutely a parallel to that. And so while we're the future of operations, at least in the way we're going with the cloud era, is invested in this whole paying for computing on somebody else's hardware. But they're gonna bill me for either by minute by the process, what have you, I want the biggest bang for my buck out of whatever I've selected for how efficient the underlying system can run. Does that make sense? And that would probably be my, the only, actually, that's what I'm really most interested in. But that's not all of the systems, that's maybe 10% of them, you know, maybe ones that are, you know, low latency. You know, a lot of events over.

Alex 57:27

So, how many languages do you speak?

John 57:32

As many as I want, as long as I can understand them.

Alex 57:37

I speak three languages. And I understand, I understand, like, one, so it's like three and a half languages, right? So and

John 57:45

You speak three languages, but you understand 1?

Alex 57:48

I understand one but I don't speak it. So it's like three and a half. Oh, I see. So you can and I used to use the language when it's appropriate. When I go to a certain country, I use that language, because I can speak it. I didn't speak English in that country, because people in that country don't speak English. Same with happy like programming languages, he used the language when it's appropriate. So you wouldn't rewrite everything and go or rust, but you would pick a component that needs as you can, you want to kind of squeeze that performance and you want to optimize it so much that you would like to, like choose rust, or go to just take everything out of that software and kind of have like the max performance out of it. So yeah, kind of I think this way is like pick the right language for the job,

John 58:36

The right tool for the job. And sometimes, sometimes it's a decision you have to make to pay off some tech debt, and you have something that's just not meeting its performance requirements, maybe you need to chop it out or create a new service out of something. And one of the determining factors of how you're going to address it might be the language you use. See, to me, that's, that sounds like a combination of a conversation I can have with a business team to talk about. I can connect performance with dollars and cents, but to justify like a complete overhaul.

Alex 59:17

That's just kind of going back to the example of migrating from Angular to React and work with products. And that's why the product was so excited about the migration because there was some value for them the react in this case, not the language, but the library, a client-side library, it was more appropriate for that. For that reason, for that purpose is to get the performance got the velocity to reuse some of the components on the front end. So it was just so so a perfect fit for that specific use case that, not only engineering but also product was excited about the migration. So you need to find that kind of match between like what's the pain point what problems and opportunities you're trying to solve and create? And what would be the right tool for that job? Is it maintaining what you have already and investing that knowing that long term, it just not gonna stick around long term is just not going to cut it. Because in the future, we're gonna have even more demands even more opportunities that we need to solve. So we need to think about a future like, do we maintain what we have now? Or do we start the migration plan and strategy and kind of piece by piece, we start migrating into something different that will give us more opportunity long term.

John 1:00:38

Yeah, I agree with that. How do you start? So it's not technology is kind of a is literally a black box for a lot of folks. And for having a mixed bag of stakeholders. How do you how do you navigate? Or what are big? So we talked about predictability, cost of new features, bugs in production? predictability, scalability, I guess performance? What other kinds of factors or leavers do you use to communicate the value of pain, technical debt?

Alex 1:01:16

Did we talk about stability? Like if you're in your system breaks in production all the time? It's a problem. Right? A lot of the time, even though we like to talk about test-driven development in the value of unit tests and integration tests, from my experience, it's rarely the case. One of the problems in our industry is deadlines. tight deadlines are always a problem. And if given the tight deadlines, people gonna cut corners. It just, it just, it happens all the time. That it's like small companies, big companies, any company testing is the problem.

John 1:02:00

I agree with that. But I also think that it's necessary. And I and I get that whenever it's like, oh, and I need you to build this feature or work on the set thing. Great. When do you need it by four weeks from now, three weeks from now, that initial anxiety sets in? Mostly because even though we know how the tools work, and how everything is set up, there's still discovery that's yet to be had. And that's the fear of the unknown, you can't help but consider that. And yes, you will cut corners. But I've also been, I've also been motivated by a deadline, a deadline has helped me understand what are the what can I fit into this increment in a way that's going to meet the immediate objective. And that's helped me largely make sure that I get something over the line that a customer can actually use, versus coming to the end of a certain period of time and saying, like, we went through this before, right, with, um, with classmates, it was when can we get to a point where people we can start getting people using this thing? And how long did that end up being?

Alex 1:03:15

I don't know, five years.

John 1:03:22

I mean, it wasn't quite five years. But it was a long time. Yeah, we spent a long, long time. And then we ended up learning something later, that we weren't even, we had so many features built-in thinking that they were going to be necessary. And they largely weren't, the client was like, hey, just give us you know, if we have just like a box where I can drop things into and go organize them, that'd be enough for me.

Alex 1:03:48

Well, it's a bit different problem conversation of like, when do you actually ship your product. So it's important for early-stage startups and companies who are trying to launch something new. But when you're actually dealing with established business and product, and you have this deadline of like, okay, we're going to have a press release or marketing campaign launched in three months. And they give the engineering team a quarter to build something. And in order to meet that deadline, engineers cut corners, and they, just the software, the code is so sloppy and no unit tests just to meet the deadline, that's a problem. So, sometimes slowing down is actually we'll get you further right over time. Sometimes going to like slowing down and taking things step by step and addressing technical debt along the way writing unit tests will get you further because the code will be more stable. It will be covered with unit tests. It will be less complicated and will be properly built-in architected. So yeah, stability, back to stability. stability is one of the red flags when It actually tells you there is a, there is a technical dead.

John 1:05:04

I agree with that. But I would also add that those practices like writing tests, for instance, give feedback back to the people building the software faster, so they can act on it sooner. And a lot in a lot of ways, it's much, it's less cost to the bottom line, to address them address issues upfront than they are to go through the whole ceremony of getting the thing into production. Having customers hit the thing and having your customers discover the problem, then you're chasing an issue. And now you're completely disrupting what you originally set out that you thought you were going to spend time on. And now you have to go back, right, you're not going forward anymore, you're going back. That's why what I like so much about I'm using Pivotal Tracker, is when you have a bug, it's a detractor to velocity. And sometimes when I'm when working with folks, they'd like to point bugs. And I was like, no point, you don't get credit for that. Right, you don't get credit for fixing a bug, that's the debt you added and we didn't do the right thing. And, and actually, what's interesting, what I'm seeing most recently is even at build time for doing builds, the faster we can get people feedback, that something is there something they need to pay attention to, the faster they can respond, the more stable the product becomes. And we keep moving forward. Sometimes I think we make the mistake of just by having software that does something that we said we'd like it to do is a success. And while it is you know, that's a big part of it. But if you can't flip it around and advance on it, and keep up with the demand of your customers without constantly paying back, right, or being chewing on debt, or chasing bugs or dealing with escalations in the middle of the night. And while you can never get rid of all of those things, if you don't have that healthy balance of keeping up with them and establishing a not perfect, I don't think you can ever have a perfect working culture. And I don't think that's the right like 100% test coverage, I would argue is a myth. And we can talk about that another time. And there's, you know, finding the real dollars and cents value out of that, but hardening you know, the things that you know, we're going to be living long periods of time that you depend on a lot and do are responsible for maybe a large majority of the compute that's happening in a given system. Those are the places that you really strive for those types of metrics. But aside from those edge cases, and then and definitely take when I say edge cases with a grain of a big grain of salt. Those are the that's where I think the value is and also keeps that feedback loop coming in from for the people that are building the stuff and getting them the information they need sooner. Protecting your customers from the human errors that we tend to create. And that's just going to be natural.

Alex 1:08:06

I mean, that's kind of the definition of technical debt. Like you skip unit tests, you borrow from future productivity, you actually harming your productivity, because you're going to have more issues in the future.

John 1:08:18

All right, Alex, any closing thoughts?

Alex 1:08:21

All right. So any other advice we can give to think about technical debt or minimize technical debt. Anything else you'd like to add?

John 1:08:31

I would I mean, yeah, I think I think largely looking holistically over where programmers are spending their time because that's who that's where the debt you can see a lot of the debt, the understanding your what kind of defects are hitting your customers? How is your practice giving feedback to your programmers sooner? And what's your How are you able to communicate are establishing a connection between the debt and your software to your product and business teams. And then from the product perspective, having an open mind to the development staff that there are problems that you might not be able to see and to ask questions, and keep an open mind to how paying tech debt could ultimately affect your bottom line.

Alex 1:09:18

Yeah, and I would like to add that use business requirements as an opportunity to align with business and product to align with your technical goals don't go against business requirements and business initiatives go with them it will be much easier to go together hold hands go together sing-song. Yeah, it's much easier to get buy-in from leadership, from your business partners to invest just a little bit extra to address technical debt as you build those features. And we talked about that with Frank in Episode Two. It's like just use business. As your friend in these conversations so that and also I have a lot more covered in my blog posts on Pragmaticlead.com so go there to PragmaticLead.com and you can read more on that.

John 1:10:14

Yeah absolutely. Also, we did plug a number of other episodes and content related so for those listening if you're interested, you can find those resources either in the show notes or on pragmatically comm or anywhere we publish a podcast. All right, Alex, thanks for hanging out, man.

Alex 1:10:32

All right, thanks. Alright, see you later. Bye.

John 1:10:36

Thanks for tuning in to the pragmatically podcast. If you found this conversation interesting or helpful, we would appreciate your feedback. If you want even more content like what you just heard. Check out pragmatic lead comm if you have a story to tell, send an email to pragmatic lead@gmail.com and someone will be in touch. Thanks again.