Redefining Engineering Productivity with Hatica

A few months ago, McKinsey went live with their report: Yes, you can measure software developer productivity. The report got everyone talking, and asking – Can developer productivity be truly measured? If yes, how to quantify productivity for engineering teams without robotizing devs, or instilling the fear and anxiety of being micromanaged and monitored every work second?

Over the years, developers have been more of investment centers than cost centers for businesses. The value proposition engineers bring to the growth of a company is unparalleled – they lay the groundwork for shipping products, building MVP to a fully functional sellable product, user acquisition, product adoption, and finally, customer retention while keeping companies ahead of the curve with constant innovation, and providing the skill to adapt to the changing market dynamics. It is practically difficult to imagine a great product without a great engineering team who knows what their customers want. 

However, this has put the engineering leadership under constant pressure of: 

• Measuring the effectiveness of their engineering teams

• Finding ways to build more productive and happy teams at work, and 

• Constantly monitoring the efficacy of any such productivity programs and practices.

Despite all the benefits developers and engineering teams bring to the table, measuring, and improving their productivity has been a black box for most engineering leaders. But if you ask me, what got us here is not getting us beyond this point if we are not empirically deliberating upon Engineering Productivity. 

Can you imagine - every single team in an organization - sales, marketing, even HR is backed with data but when it comes to engineering teams there is sudden loathing that, “We create, our work cannot be quantized”. But I do not believe that. Would you expect me to go tell a salesperson that there is no “art” in selling or a marketer that there is no “magic” in marketing? Then you cannot bargain the lack of data for just being a developer or engineering team!

 As I always say; 

“What cannot be measured cannot be improved. Setting up your engineering team for success is a double whammy because neither productivity can be measured directly nor the ways to monitor and improve engineering productivity are foolproof yet!”

In this blog, I try to highlight the nuances of measuring engineering productivity and how to work on a more agile framework that could suit your business needs irrespective of how big your engineering teams are, how they are structured, and what unique challenges they straddle with.

Measuring Developer Productivity: The Old Ways Don't Work Anymore? 

Well, it all begins with gathering data. With the SaaS sprawl and a huge tech stack, gathering data can be challenging, difficult, and almost impossible at times. But there is no escaping the fact that one has to start somewhere to track, monitor, and improve the overall engineering productivity for an engineering team. The question engineering leaders need to ask themselves is, “What do we measure in the first place?” Because not measuring is surely not the solution.

We all agree every engineering team is different, and so will be their methodology for measuring productivity. The relevancy of the data points will vary wrt. individuals, team setup, or organizational structure. On top of that how we leverage these data points or key engineering metrics is another problem to solve. 

“When a measure becomes a target, it ceases to be a good measure.”

While some engineering leaders use DORA metrics to gauge the effectiveness of their engineering efforts, others still rely on more traditional indicators like LoC written, bugs fixed, story points completed, or number of features released. 

The approach keeps varying depending on sprint goals, an engineering team's size, structure, and delivery commitments in a given period.

But in many cases, I have seen that the desire to monitor and measure productivity translates into immediate bossware practices which are mostly counter-intuitive and can even sabotage the overall developer experience. 

The idea of tracking and monitoring activities across projects across the software development lifecycle might sound fancy yet it offers an incomplete picture and comes with a highly skewed lens. 

While DORA metrics are great metrics to track, these four metrics unknowingly narrow down productivity to just measuring “software delivery” without capturing other dimensions like coding practices, commit cycles, soft metrics–team alignment and team dynamics, let alone uninterrupted time developers get to code, developer burnout, and code maintainability.

The dangers of resorting to vanity metrics are already well-known. These indicators narrow down productivity as just a measure of developer activity (rather than a wholesome preview into engineering productivity)– as in how fast the features have been delivered, or how much positive work has been performed. Though these metrics serve as a litmus test paper for development work, they limit developers to just a technical resource. More importantly, they limit the ability of engineering leaders to get a 360-degree view into how their engineering teams operate and function and how this activity is going to translate into results in a long time. 

For instance, the concept of using ‘the number of features released’ metric sounds progressive. There have been times when I have felt this urge to look at these vanity metrics for a given sprint but I have realized this is a shallow gratification I can greet myself with as a leader. I have dealt with very similar kinds of concerns and challenges in my previous roles as well before I took it upon myself to resolve this for engineering teams globally. 

A developer might have released several features, but if they're all of low quality, then it’s just a dent to overall UX and dev effort only resulting in more rework. On the other hand, a single, high-quality feature might shoot up a team’s customer experience exponentially.

This traditional approach vehemently turns a blind eye to the modern way of work where developers work more collaboratively by jamming with engineering managers, and product owners to plan, and scope iterations, deploy code with DevOps teams, run regular sanity checks with QA, and help SREs to ensure KTLO. The engineering teams today are more inclusive than before.

“Modern engineering teams thrive with inclusive frameworks to push higher engineering productivity. It's not just fair, it's faster.”

This increasingly interconnectedness is often missed in the traditional framework alone. 

The Four Pillars of Engineering Productivity

While our fascination with developer productivity has been more or less constant for the past years, our understanding does keep evolving. 
The advent of the SPACE framework shifted conversations around how engineering leaders measure, and think about engineering productivity. It was the first hardcore research that helped fetch attention from measuring just “activity-based, easily measurable signals” towards the most overlooked aspects of how engineering things work, and deliver, including developer well-being, communication, and ability to get things done without delays or interruptions.

This framework highlighted the gaps in how engineering leaders approached developer productivity, thereby paving the way for a new age model to resolve challenges of measuring engineering effectiveness. 

After having solved the problem of workflow visibility and getting actionable insights for more than 600+ engineering teams globally, talking to 100s of engineering leaders of small, medium, and large enterprises - we have understood a lot of engineering teams share their concerns. Most of them know that something is off but either they have no visibility into developer workflows or are completely blindfolded to engineering analytics. Some of them who have figured out what to solve are not sure if the same fix will help in the future as well. 

As a result of these deep-dive discussions, mapping these causal factors onto the larger picture, and tying these back to how we at Hatica solve for higher productivity with actionable insights - we have figured out that this often translates into four key buckets which we call The Four Pillars of Engineering Productivity:

• Team alignment

• Delivery velocity

• Software quality

• Developer Experience 

To better understand the four aspects, let’s take the analogy of a road trip. Let’s picture this: you're cruising down the Pacific Coast Highway, heading from San Francisco to LA. Reaching your destination efficiently in time, without many bumps, requires a balance of four key factors:

• Here alignment ensures you don’t just drive aimlessly, but in the direction of LA. You have a clear route mapped out, everyone in the car knows the destination, and you're constantly checking the GPS to ensure you're on the right track.

• An optimal velocity means you are driving at a steady pace, close to the speed limit, but prioritizing safety and most importantly you are headed in the right direction hence velocity not speed. 

• Quality ensures you're not just focused on getting there quickly. You're attentive on the road, avoiding potholes and potential detours so your trip is more memorable and less nightmarish. 

• Focusing on health means you are not pushing yourself to exhaustion. You are constantly alert, take regular breaks to cool down after a tiring day, and find time to spend with your family - implying your overall experience of reaching the destination matters as much.

That way, you reach your destination on time, happy, and refreshed! 

Isn’t it very much the same as the process of shipping software? Now that we had the perfect metaphor of a breezy road trip, here’s a ground-level view of the four pillars, and their respective metrics when it comes to building and shipping software: 

Team Alignment 

The first and foremost pillar ensures your engineering efforts are aligned with business expectations, and the bottom line stays afloat. Team alignment is a rich parameter for leaders to estimate the ROI of engineering inputs, understand where engineering spending is being made (roadmap vs KTLO vs bugs), and visualize the value created by each dollar spent. 

Metrics for Team Alignment 

• Resource allocation per work type visualizes the distribution of engineering effort into feature development, technical debt, and unplanned work. The metric is a breakdown of where your developers are spending their time, and whether their workload is skewed towards non-roadmap tasks. 

• Resource allocation by project metric for a 360-degree view into engineering investments made on business-critical, and shadow projects. 

This way an engineering leader is in a better position to see and estimate the efforts the team is putting into various projects and stay abreast if they are aligned with the overall business goals of the organization. I believe even if you are working day in and day out but that work is not aligned with the overall business goals, that work is of little value. This is where deep diving into investment and resource allocation gives you leverage to move resources across projects and also plan/ship projects likewise.

Once you have figured out the alignment of the engineering team and effort, the next comes the velocity at which the code is being shipped. 

Software Delivery Velocity

If data is the new oil, time is the new currency, especially for engineering teams. They have to be constantly time-aware to deliver on timelines and innovate fast. 

But this can happen only when we know how our teams operate, where they are stuck, what’s blocking them and the next plan of action to ship faster. 

As an engineering leader, are you equipped to determine how fast your developmental pipeline is moving, and the pace of innovation? The 2nd pillar called Delivery Velocity is crucial as it helps: 

• Predict a feasible software delivery timeline, understand delivery trends (spikes, and dips), and time taken to develop a feature 

• Set realistic client expectations and back work delivery velocity and timelines

• Expose engineering inefficiencies and bottlenecks in the development pipeline to meet the agreed-upon timelines

From an insider's view, it could mean improving PR review times so they don’t delay releases, and leave sprints incomplete week over week, improving commits in a week, or optimizing engineering operations to reduce the overall cycle time.

Key Metrics To Estimate Delivery Speed 

• Cycle time metric estimates the amount of time it takes to complete a piece of work, right from writing the first line of code to shipping it to production. The signal is crucial to visualize engineering velocity and understand where your developers are blocked in the pipeline. It gives you a thorough understanding of your developer workflows and how your engineering team operates.

• Track coding time to ensure your developer spent enough time on their IDEs to write quality code, then focus on non-core tasks like avoidable meetings. High coding time can help you flag burnout tendencies, unstable workload distribution, and the growing complexity of the codebase.

• A look at the code review, and rework time through the lens of metrics like pickup time (time between raising a new PR, and receiving its first comment/reaction), and review time to merge helps diagnose roadblocks in the code review process: uneven distribution of review tasks, lack of quality reviews, large PRs, or organizational silos. (The snapshot above is a view of one of Hatica’s dashboards that allows you to look into these metrics across teams and projects with precision thereby helping you make decisions using actionable data points)

• Analyzing time to deployment can help the team understand if they are facing long wait times, or flaky builds that delay merge and deployments. 

Net-net, Alignment gets you direction, velocity gets you timelines - it is important to ensure that the quality of the software being released is not compromised. 

Software Quality 

I have seen it firsthand with very able engineering teams failing to ship quality products because they were hooked to the idea of shipping before time. At times, either the use cases were not thought through enough, or the code was too buggy to fail later. 

Fast is meaningless if it's broken. Bug-ridden software is a productivity sinkhole. It wastes engineering resources and efforts, delaying features, and frustrating users. 

The idea is to deliver quality, stable products without breaking things in the process or using too many quick fixes that later increase your technical debt and crash rate. 

So in fast-moving projects and work requirements, it becomes important yet difficult to ensure the quality of the code being released.

Key Metrics To Measure Software Quality 

• Change the failure rate to highlight the effectiveness of the testing process by analyzing the percentage of deployments causing issues requiring rollbacks or fixes. 

• MTTR as an indicator of software resiliency, and how fast it recovers from service disruptions. 

• Bug filing rate to measure the frequency with which bugs are reported in a software product.

Developer Experience

Talking about developer experience, we must pay due attention to both people and process health. Prioritizing developer well-being and maintaining a healthy engineering process should be the first step toward building a high-performing engineering team. 

On my journey from Head of Engineering at Uber to Founder at Hatica, I have taken this as my personal goal to ensure that my teams are healthy and happy while being at their productive best. This has been possible by leveraging data not to micromanage but to understand how my teams are doing at and off work. 

You remember the road trip analogy I talked about initially: what if there were no breaks, no cooling period, or focus on snack breaks during the trip? While you might have reached the destination, would it have been an equally meaningful, effective, and enjoyable journey? Probably not.

Trust building, recognition of engineering work, sustainable workload, a culture of psychological safety an antidote to burnout, and even faster software delivery– go a long way in creating a positive developer experience. 

While this was not an important factor a few years back while engineering leaders forayed into building stellar engineering teams, today it is the core philosophy behind driving engineering excellence. Happy and Healthy teams drive stellar results!

Equally important is to maintain the integrity of engineering processes to continuously improve engineering operations, forecast project delivery with accuracy, and ensure PR hygiene, so the team can ship sprint over sprint. The method drives the madness better.

Key Metrics To Gauge People's Health

• Tracking, and acting on maker time can help engineering managers to minimize distractions, reduce cognitive clutter, and improve the focus time of their direct reports. It is also directly correlated to the effective coding time of your engineer. As someone who has coded for most of his life, I cannot emphasize enough the need for uninterrupted time to code.

Hatica’s Maker time dashboard allows you to deep dive into work patterns and improve it for better maker time available to your developers: 

• Quiet days to understand the pattern of engineering activity after office hours. It is also a chance for EMs to reflect whether the workload on dev is unsustainable or preempt burnout tendencies.

• DevEx surveys capture developers’ expectations, feelings, opinions, and comfort with the current engineering paradigm. It’s an important perceptual metric that helps leaders to listen to ground-level devs, and construct a broader picture of engineering effectiveness, and productivity. 

Key Metrics To Understand Process Health

• Planning accuracy is an indicator of unplanned, and ad-hoc work taken by an engineering team. It might also mean some other barrier, maybe developer turnover, high context switching, poorly scoped iterations, or even accrued technical debt. 

• Code churn measures the amount of code that is being changed over time. This metric can help engineering teams pinpoint areas where they need to prioritize refactoring. 

• A potentially high number of unreviewed PRs merged is an indicator of a lack of sanity checks during the review process, and even increases delivery risks, and bugs associated with the release. We aren’t talking about the implications of technical debt yet!

• Bigger PR size means that the features are not broken down into individual units, and the code review might turn slow, or even inefficient. 

These metrics are your engineering system’s tuning forks and not whips of micromanagement. These metrics are meant to be a means to an end of high engineering effectiveness, reduced wasted effort, and a positive developer experience, but over the past few years, engineering metrics have often gotten mistaken for an end in themselves. 

That’s why today engineering teams have a lot of inertia towards looking at these metrics. Putting these metrics into the wrong use only defeats the purpose of what they are stitched together.

It is important to set benchmarks for your team internally and externally (for collaboration) so that speed is not taken as an alibi for shipping poor-quality products. Engineering teams need to be told how alignment-velocity-quality only results in more productive and efficient teams.

(Read more on A Comprehensive Guide on Self-Assessing Engineering Teams)

What’s important is to understand that all four pillars are equally important, and no one precedes the other. Can you prioritize speed over safety? Or, choose to keep driving ruthlessly and risk an accident? We already know the answer. 

Similarly, engineering leaders cannot maximize productivity by focusing on activity alone, or deprioritizing developer well-being, especially when there is this brimming sentiment that a growing obsession with velocity as a north-star metric can translate to higher-tech debt and even developer burnout. Looking at any of the pillars in isolation and not in tandem with others is like looking at your engineering ecosystem with a narrow, fractured lens. 

And that is why we at Hatica have attempted to redefine engineering productivity with a balanced perspective. We sensed that while a lot of developer tools promised several metrics that engineering leaders could skim through the dashboards, they lacked the vision, narrative, and a strong purpose behind bringing the numbers together. 

Our attempt to bucket these metrics into The Four Pillars of Engineering Productivity is a step ahead in the direction of developing a holistic outlook toward building high-performing yet happy/productive engineering teams.

Continuously finding a balance between the four pillars is not only going to help us produce a healthy engineering culture but also strive for engineering excellence as a continued goal. 

If you are also looking to improve the overall developer experience and boost engineering productivity for your team, you can set up some time with our productivity experts who can guide you through the next steps with Hatica!