The Optimal Customer-Driven Release Frequency Formula

Summary

  • Why does understanding optimal release frequency matter?

    • Understanding and acting upon what drives an optimal release cycle ensures that the value your team delivers is ultimately absorbed and utilized. 

    • Delivering value at a pace that stakeholders are unable to utilize is only marginally better than not delivering that value at all, with the marginal value difference consisting of the hope that those stakeholders will utilize that value at some point in the future. 

  • The Optimal Customer-Driven Release Frequency Formula

    • Determining how frequently product and engineering teams release changes can be a contentious topic within organizations, but it should ultimately be based on two values, which are themselves multi-factorial.

    • The following formula optimizes for product success over the long term, and while release cycles can be based upon other factors, those factors are typically short term/single objective focused instead of process focused. 

    • At the highest level:

      • Optimal Release Frequency = (Average Time Required for Internal Stakeholders to Absorb Releases + Average Time Required for External Stakeholders/Customers to Absorb Releases)

        • Ex = 6 weeks for Internal + 6 weeks for External = 12 Week Release Cycles

        • Note - Sometimes these activities can be done in parallel.

      • Average Time Required for Internal/External Stakeholders to Absorb Releases are each composed of time required for [Training + Documentation Updates + Usage]

    • Full Formula 

      • Optimal Release Frequency = 

        • Internal Stakeholder Training + Internal Documentation Updates + Internal Usage

        • External Stakeholder Training + External Documentation Updates + External Usage

  • Incorrect Drivers of Release Cycles

    • Release cycles are often principally driven by a number of reasons other than the above, which are sub-optimal in a number of ways and explored further in the discussion section. This doesn’t mean the following factors shouldn’t be considered, simply that they should not be the key drivers over the long term.

      • Internal Process - Agile says we should release every 2/4/X weeks, so that is when we’ll release!

      • Publicity/Marketing/Sales / Event Driven - We have a quarterly/semi-annual/annual conference coming up, we need new features to talk about!

      • Staff - We have enough staff to release every 2/4/X weeks, so that is when we release!

      • Competitors - Our competitors sell their customers on new features every 2/4/X, so we’ll do that too!

  • Optimal Customer-Driven Release Cycle Cheat Sheet

    • Includes formulas and factors related to the optimal CDRC (PDF), (PNG), (PPTX).

Detailed Discussion

Far too often, product and engineering teams will ship features and releases at a pace that is inconsistent with the goal of delivering the maximum amount of value to both their internal and external stakeholders. This is surprising in the sense than good product teams are theoretically the “voice of the customer” and are responsible for understanding what solutions will deliver the most value. Ultimately, the ideal release cycle delivers value at an organization-specific pace that internal and external stakeholders can maximally absorb and utilize. Analogous to the shift towards customer driven development, we focus here on shifting closer to customer-driven release frequency.

Just as it is critically important for product and engineering teams to identify the right problem and the right solution, it is equally important that those teams ensure customers and stakeholders are set up to successfully utilize those solutions upon delivery. If the release cycle is too rapid, customers and stakeholders will simultaneously be frustrated at the rate of change and their inability to learn how to leverage the new features. If the release cycle is too lethargic, customers will churn and stakeholders will yearn to play on an innovative team. 

Finding an ideal release cycle involves exploring a number of different drivers that are specific to your stakeholders and your customers, which are primarily driven by how quickly those stakeholders can utilize the value you create. These drivers converge on an optimal customer-driven release frequency formula. In addition, there are influencing factors that should be considered in determining your ideal release cycle, but should not be driving its pace over the long term.  Let’s explore the drivers that make up the optimal release frequency formula alongside each of the primary influencing factors. 

  • The Formula

    • Optimal Customer-Driven Release Frequency = Average Time Required for Internal Stakeholders to Absorb Releases + Average Time Required for External Stakeholders/Customers to Absorb Releases

    • Average Time Required for Internal/External Stakeholders to Absorb Releases are each composed of time required for [Training + Documentation Updates + Usage]

  • Drivers

    • Average Time Required for Internal Stakeholders to Absorb Releases is composed of:

        • Training - Is each internal department aware of the new changes and how to support customers?

        • Documentation Updates - Is each artifact used by the organization properly updated to reflect the new changes?

        • Usage - Are employees using or interacting with the new changes? Have they incorporated them into their workflows?

    • Average Time Required for External Stakeholders/Customers to Absorb Releases is composed of:

        • Training - Do customers understand how to use the new features? 

        • Documentation Updates - Is each artifact used by the customer properly updated to reflect the new changes?

        • Usage - Are customers using or interacting with the new changes? Have they incorporated them into their workflows?

  • Influencing Factors

    • Internal Process - Sticking to a release cadence or process that delivers change at an arbitrary, albeit consistent, time interval is largely unhelpful if customers and stakeholders can’t absorb the delivered rate of change. Many teams marshal continuous delivery or two week release cycles as proof that they are delivering significant value to the business without discussing or disclosing how much of what was shipped is actually being utilized. Shipped Value != Absorbed Value.

    • Publicity/Marketing/Sales / Event Driven - There is absolutely no product and engineering team on earth that can avoid the occasional release that is scheduled principally around a specific event or client. There will be big clients your company needs to win and features built specifically for tradeshows. It is important to accept that these situations will occur, understand that these events are exceptions to your ideal release cycle, and explain to stakeholders why these occasions are in fact exceptions to the optimal customer-driven release cycle your team has so carefully determined. 

    • Staff / Empire Building - The size of the product and engineering organization can be a factor when it is too large or over-resourced relative to the products it owns. As discussed previously in Empire Building, product and engineering teams will grow over time (Great!), but the products they are responsible for do not necessarily need to grow or change at a similar rate. In an ideal world, those product and engineering resources are reassigned to new products. In the real world, those resources stay put and justify their existence by continuing to release new features that are unnecessary at worst and marginally helpful at best. In this case, the optimal release cycle has likely shifted and product/engineering resources must shift as well. 

    • Competitors - As software consumers have become more aware of the SaaS business model, rate of value delivery and agile development have become points of differentiation in the sales process. This is unlikely to change at any point in the future as it would require massive customer re-education by sales teams that are incentivized to do precisely the opposite. It is your responsibility as the product and engineering team to educate stakeholders on the rationale behind your release frequency and the data that it incorporates. Do not assume that even though a 6 month release cycle is ideal for your customers that those same customers will automatically admit to that fact or appreciate your rationale. 

Examples & Suggestions

While your release cycle will be driven by the values specific to your customers and stakeholders, there are several rules of thumb that can be helpful, all subsequently discussed as well as summarized in the diagram below.

  • Product Complexity - As product complexity increases, the length of your release cycle will likely naturally increase. This is primarily due to the multiplicative factor of product complexity, in that for each required change management activity typically required for a new release (Training, Documentation, Usage Monitoring), each of those activities will increase in proportion to the complexity of the product. An example of this can be seen in comparing the Uber application to Facebook - both are consumer facing applications, but Uber has a significantly narrower consumer facing scope than Facebook, which allows more frequent changes around that scope.

  • Customer/Stakeholder Dedication to Product - Release cycles can be shortened if you have a fanatically dedicated customer base, either due to a natural monopoly or the criticality of your software to your customer’s tasks. Dedication matters because it will increase the rate at which customers can and want to utilize the changes that your team ships. A great example of this is Adobe, whose products are all mission critical to the media production industry and are natural monopolies in their space. 

  • Customer/Stakeholder Industry Regulation - If your industry is heavily regulated, there may be a maximum rate of change that your customers can legally or logistically absorb. This is nearly zero for consumer facing applications whereas it tends towards infinity for nuclear power plant operating systems. This point is less a commentary on whether the regulation is justified or not, and more focused on determining what effect it will have on your release cycle. 

  • Dedicated Change Management Resources - If your company has an appreciation for change management and appropriately staffs training/documentation resources, it may be possible to dramatically shorten your release cycles. For example, if your customer success team is running a webinar on a regular basis to educate customers on upcoming and already released features, your organization can realistically ship change at a much faster pace than if change management is primarily handled by sending out educational emails that customers and stakeholders don’t read. 

The main objective is to ensure that the value that you ship is approximately equal to the value utilized by customers and stakeholders. If it isn’t, resources are being allocated in a sub-optimal fashion, and your team should consider changes to one or more of the variables discussed above.

Product Management Due Diligence List

Summary

  • Conducting due diligence is a skill typically relegated to consulting and investment firms, but it is equally as critical to product managers when evaluating products that they may decide to buy versus build or partner

  • While due diligence typically encompasses all divisions at a given company (financial, legal, sales, marketing, engineering, etc.), we’ll focus on conducting diligence on software products from the perspective of a product manager. 

    • Note that despite this being a product focused list, we will necessarily wander into other divisions of the company due to the nature of product management’s widespread interaction and influence. 

  • This list aims to be as comprehensive and exhaustive as possible, so as to be useful not only during diligence but also as a check for product teams as they build products to ensure they are thinking objectively about their product’s success.

  • Even though lists are difficult to summarize by their nature, the key question to continue to ask about the aspect of each product during diligence is simple - why?

  • Similar to the 3 Levels of Product Thought, diligence can be thought of similarly:

    • Phase 1 - Discover the Current Product 

      • The objective in this stage is not to form an opinion, your role is that of a detached and curious observer intent on understanding the product in its current form. 

    • Phase 2 - Understand the Product’s History

      • The objective in this stage is to understand why the product acts / feels / was designed / was priced / was coded / was released / was sold / was marketed in the way that it has been. 

      • Historical product choices were made for a reason, those decisions and their rationale may be illogical or irrelevant, the goal in this stage is to understand how and why those decisions were made, and begin to form an opinion as to whether those were the right ones. 

    • Phase 3 - Form an Opinion on the Product’s Future 

      • The objective in this stage is to incorporate your understanding from Phase 1, and your historical research from Phase 2, in order to form an opinion on the product’s future.

      • Just as we asked a series of questions in Phase 1 and 2, we will now be providing our opinion in order to answer those exact same questions. 

      • These answers will help us determine if this is truly a product our company wants to purchase. 

  • Downloads - I’ve converted the framework and question lists into downloadable PDF and Word versions.

Detailed Discussion

As the overall due diligence framework was discussed in the summary, we’ll use the detailed discussion section to break down in further detail the exact questions I would, and have used, to evaluate products for potential acquisition. 

The framework above is how you should be thinking about and evaluating the information that you obtain with the following list of questions. Ideally you are using the following questions to build your layers of understanding, but you’ll ultimately need to organize the information provided in to the framework above as it is almost always provided in an ad hoc manner

I have historically tried to pre-solve this problem by structuring my questions around each phase, asking one after the other, but product management is complicated, and answers will frequently interweave information that will be valuable at each stage of your understanding. 

Given the reality of the diligence process, your role as a product manager or analyst is to first capture all the information required and then organize it in to the above framework. In my experience, it is virtually impossible to ask the following list of questions so that the verbatim output and answers are neatly formatted and analyzed. Organizing the information is half the value provided during diligence. 

  • Product Management Due Diligence Question list

    • Customer Observation and Interview - Minimum 10 customers. 

      • While it can sometimes be a challenge to convince the company to let you sit with their customers, it is absolutely critical to understand from a customer perspective how that product is utilized. 

        • One “hack” around the difficulty in obtaining customer access is to watch YouTube tutorials of the product from individuals not associated with the company. These tutorials will often illuminate the flaws in the product that only users will uncover. 

      • Ideally, you interview and observe customers from each of the customer personas and segments discovered during diligence, this can help your team understand where potential opportunities / gaps remain. 

    • Customer Product Experience

      • In addition to watching the customer use the product, it is important to understand their qualitative experience about the product as well. 

      • Feelings

        • How does the product make them feel? Efficient? Happy? Sad? Annoyed? Ignored? 

      • Onboarding

        • How are customers onboarded? Is this automated? Why or why not? Is this the right approach? 

          • Most B2B companies have manual onboarding steps, most of which can be automated at no cost to the customer experience 99% of the time. This is one of the most straightforward product improvements to try to find. 

        • What is each customer’s time to value (The time it takes from when they are sold to when they actually derive value from the product)? Break out by persona / segment / cohort?

          • This can be surprising as well if companies have long sales and installation cycles. Massive improvements can be had in customer satisfaction by fixing this step. 

      • Billing

        • Relating to pricing, how do customers actually pay for the product? Credit card / ACH / Invoice / Check?

        • Are there any issues with billing? Are customers happy about it or would they prefer another way? 

          • You can often produce huge improvements in customer product satisfaction by simplifying the billing process. One would think that companies would be incentivized to make it as easy as possible for customers to pay, but this is not always the case. 

      • Support

        • How are customers supported? Email / phone / chat / text / social channels / in person / support system (Salesforce / Zendesk)? Are these channels good / bad / worthwhile?

        • Is there a public knowledge base? Is it used? Why / why not?

        • Are customers happy with the responses they get? If not, why? If so, why?

        • How, if at all, does customer support relate to customer churn? 

      • Contracts & Service Level Agreements

        • What is the contract signing experience like? What are sample contracts for each persona and customer segment? 

        • What are customer service level agreements? Are they profitable for the company? Are they too strict / too lax? Have they been routinely violated?

        • What are the termination dates? By cohort / product / persona / segment? Is there a termination “cliff” where many contracts come up for renewal at once? 

      • Localization

        • Are all customers English speaking? Are there localization issues? Would the product perform better if localized? Would it reduce complexity to have one product language?

    • Product Team

      • Who are the members of the team? Are they effective? What performance review process exists? Is it effective? Do you want to retain all / some / none of the product team? 

      • What is the product culture? Is it the same as yours? Is it focused on growth / stability / security? What incentive scheme does it operate under? Is the team effective? Will the culture merge with yours? 

      • How is knowledge shared and capture? How are features and product artifacts managed? 

      • Is the product team all in one office? Different offices? Are they remote or in office? 

    • Product Artifacts

      • Competitive Landscapes

        • Who are the primary competitors in the space? 

        • Who are the secondary competitors? Are they worth buying? Why are they secondary? 

        • Who does your sales team see in the sales process? Who do they lose against? Who do they win against? Why do they lose? Why do they win? 

        • Are their substitutable goods / services / products that have been ignored? 

        • Are there competitors that the company is losing against that they are not tracking? Are there competitors on the horizon?

      • Market Maps

        • Market maps are different than competitive landscapes as they include products and services that are in adjacent spaces and industries. These are important to understand both for future product opportunities as well as potential competitors.  

        • What is the space that the company operates in? Is it one of fierce competition (productivity tools) or quasi-monopoly (Uber / Lyft)?

        • Why are they in that space?

          • Most would assume that the company operates in a space that is optimal which is not always the case. Companies can long operate in business spaces and segments that are sub-optimal for a multitude of reasons, some rational, some less so. 

      • User Stories

        • How does the product team produce their user stories? Obtain examples for a recent release. 

      • Customer Personas & Segments

        • Who are the customer personas? What do the product’s customers look like? 

        • What are the customer segments? Are they the same as the personas or different? Why are they different?

      • Product Vision

        • What is the product vision? Why is that the product vision? Is there a product vision or is there just a list of features to be developed?

          • Having a product vision incorporates all of the elements discussed in this list to produce the why behind the value to be delivered to customers.

          • Many companies simply have a long list of prioritized features that are divorced from or ignore the ultimate value they provide to the user. 

      • Go To Market & Target Market Analysis

        • How has the company gone to market with new products and features? 

        • What is the rollout process? For product? For engineering? For marketing? For sales? For services? For customer success?

          • It is important to ask each of these departments independently and without product team members in the room. The goal here is to understand from each division’s perspective how the go to market process works without product influence. 

      • Pricing

        • How are products priced? Monthly / Quarterly / Yearly? Recurring subscription / one time payment / combination? 

        • Are there discounts given? Why are there discounts given? Should they be given?

          • Education / military / special segment discounts?

        • Are products sold at list price or are there discounts given? If so, what are they and why are they given? To what personas / segments?

        • How are those prices displayed to customers? Are they on the website? Do customers have to call or request a quote? How long does it take for a customer to obtain a price for a product?

        • Are the prices reasonable or should they be lowered / raised?

          • It is shocking the level of leverage companies can obtain by simply raising their prices. While this can have many effects, in general it will weed out the customers that are most costly to you / the company and keep the most profitable customers to you / the company. This occurs because raising prices forces those customers to evaluate whether your solution is really worth paying for on a recurring basis.

      • Product & Feature Priorities

        • Relating to product vision, what are the current priorities for each product and how are those broken down into features? 

        • Why are they the current priorities? How were they prioritized (Value, CEO/VP Edict, Big Customer Request)?

          • How products and features are prioritized will tell you a great deal about how products are built, how the product organization functions, and where power is held within the organization. 

      • Product Roadmap - 3 / 6 / 12 Month

        • Relating to product and feature priorities, how do those priorities come together to form the product roadmap for the next 3/6/12 months? 

        • Where is the product roadmap stored? Who has access to it? Why is that the case? 

      • Partner Ecosystem

        • Who are the current partners? What functionality do they provide? Why were those partners selected over other partners in the space? 

        • Are any of those partner solutions better integrated through either building those solutions or acquiring them?

    • Data & Metrics

      • Cohort Analysis

        • What are the retention rates for both customers and revenue, on a per product / customer segment / user persona basis?

        • If you gave me one data point about a product, this would be it. Nearly every other issue can be fixed in this list, but if you can’t retain customers and/or sell them more over time, the product has fundamental issues. 

      • Customer Acquisition Cost - CAC

        • How are customers acquired? What channels? At what cost? Are some channels better / worse than others? Are some channels not being used? 

      • Customer Life Time Value - CLTV

        • What is the lifetime value of your customers? By segment? By persona? 

        • Are all of these profitable? If so, could they be more profitable? If not, why not? 

          • This is another high leverage question, more often than not, there are opportunities to cut underperforming customer segments and raise prices on better performing segments. 

      • Product Usage Metrics

        • How is the product being used? How is that being measured? Time on site / DAU/MAU, core actions taken? 

        • What tools are used to measure this (Google Analytics, Optimizely)? Are staff trained on these tools? Why / why not? 

    • Technical Diligence

      • While technical diligence is an entirely different essay, it’s worth considering some of the questions that can affect the product. 

      • 3rd Party Resources

        • Does the team rely on any consultants? Have they signed the appropriate legal agreements (Ex - IP Assignment, NDA)?

        • How much of the product was developed / designed in house vs outsourced? 

          • This can matter if you are buying just the IP or the team as well.

      • Engineering Process

        • What process does engineering run on? Agile? Waterfall? Hybrid? Is it effective? Why / why not? 

        • Are there dedicated test and staging environments? If so, how are they used? If not, why not? 

        • What systems do you use for code collaboration and revision? 

        • How are secrets (AWS Keys, Encryption Keys, Logins) handled and shared? What are your access controls? What personnel have access to these items? An ability to change them? 

        • Does each product manager have their own engineering resources? Or do multiple product managers assign tasks to one engineering team? 

      • Data Handling, Backup, Recovery

        • What is your disaster recovery plan? Has it been tested? When was it tested? What was the result? 

        • How is data backed up? Has it been tested? When? What was the result?

        • How is data secured? Have you ever conducted security audits? If so, what were the results? If not, why not? 

        • Where is customer data stored? Cloud / on premise / hybrid? 

      • Technology Stack

        • What languages is the product built with? Does your company have expertise in these languages? If not, are you prepared to obtain them? If so, is the level of expertise the same?

          • This is a critical question, as often tech being on a different stack can rule out an acquisition entirely. 

Details Matter, Even the Subtle Ones [Good UX]

Product Background

While Apple has had their fair share of design missteps, Apple has remarkable product design consistency not only across products, but across platforms as well. 

User Experience

Most smartphone users are familiar with the “blue bubbles” that indicate whether a given message is an iMessage versus a light green which indicates a normal text message. This style is consistent across all iDevices (iPhone, Mac, iPad, etc.). More info at Apple, but image below.

This color scheme is also on display in Apple’s support conducted via web chat as seen in the image below, even down to my response “messages” being the exact same color grey as those in an actual iMessage conversation. 

This is a remarkable detail to get right, even at a company like Apple that prides itself on a consistent user experience. This design choice likely creates a higher degree of trust as well as the opportunity for the user to stay within Apple’s carefully curated user experience garden. 

Compare this to Intercom’s chat (which now looks eerily similar to iMessage) or to Zendesk’s chat platform, and imagine if Apple used Zendesk’s messaging platform for support. It would take the user out of the Apple “experience”, which even for just a moment, hurts the world Apple has created and attempts to maintain. 

Garbage In, Good Design Out [Good UX]

Product Background

I had the chance to travel to Utrecht in the Netherlands this past summer and one of my favorite examples of good design that I found there was actually a trash can and recycling wall!

User Experience

Normally, after enjoying a meal at a fast food restaurant where the patrons clean their own tables, the moment comes for each diner to awkwardly attempt to empty their tray into the trash can without touching the lid that separates the trash can from the interior of the restaurant.

This problem has been solved with the receptacle relocated to the top of the trash can to a certain extent, but many restaurants still use the old standby with the opening on the side with a vertically swinging door.

This restaurant cleverly put a handle that, when pulled, automatically opens the trash can door. This is a superior design in that it allows users to avoid touching the garbage receptacle door and instead use a lever to open the door further. This design could be improved even further if the handle had anti-microbial coating on it.

The recycling wall behind the trash can also provides a much more interesting and artistic way of recycling glass bottles than throwing them in a bin.