3 LEAPS

We help transactional businesses harness scale as their superpower.

[3 LEAPS @3LEAPS]

New threads, broader mission: helping transactional businesses and their partners wrangle their scaling problems into submission. We've leveled up our game to turn your business challenges into solutions. Want to see what we're cooking? Hop over to our new website at https://t.co/LWr24NVqQG. Call or message us - we would love to explore how we can help you! #3LEAPS #ScaleUp

[3 LEAPS @3LEAPS]

Smooth Development Maxims - Build In Observability Part 2

Practical tips for improving scalability and resiliency with observability

Here are some tips we've accumulated from work with clients across a wide range of projects. DM me @3leapsdave or seek help from an experienced architect if you aren't sure how to get started.

1.) Log to STDOUT as a general pattern, using a well-supported library to format and direct traffic where needed. Use STDERR as well if you have a consistent pattern and a good reason for separating.

Code is cleaner if you always write the same way. It is hard in large systems to keep up with file pointers.

2.) Implement centralized logging with an agent - being careful to avoid blocking

You'd be amazed how many times we have found applications that will literally stop running if the log queue backs up.

3.) Write logs in a structured form using JSON or other notation

Yes, it is harder to read on a console. However, having real "key value" patterns and consistent sets of data from each message means your analysis tools on the collection side will work much better. Remember, logs MOSTLY exist to sort out an issue AFTER it happens. The easier you make the forensics process, the faster you will find your problems.

4.) Instrument your code with event and place marking telemetry

Libraries like OpenTelemetry provide a simple pattern to enable the code to say "I'm here" . For larger applications, it often makes sense to have some telemetry streams that are separate from the main activity log. We prefer this pattern for marking state in complex back-end operations, as an example.

5.) Select and use an analysis and visualization system for your logs and event data

Some like the ELK stack (Kibana being the visualization part). Some like Grafana + Prometheus. There are many excellent open source commercial solutions that integrate analytics and alerting for logs and events. If your solution has large data pipelines, think in terms of a "Data Operations Console" that provides real-time information on data flows and issues.

As with all refactoring exercises, a key to success is choosing a balanced mix of "small quick wins" and "big winners". As we wrote in Part 1, find the blind spots first and fix them. Start small on your alerting and analysis with one key area - get that right and then add more. Dedicate a few stories each sprint to improving telemetry.

Measure, Monitor, Manage!

#smoothingsolutiondelivery #measuremonitormanage #3leaps

[3 LEAPS @3LEAPS]

Development Maxims - Build In Observability

Measure, Monitor, Manage

While I am not certain that everything about software development is improving over time, it has never been easier to monitor the performance of applications. I started working in a time when we flashed LEDs or seven segment displays for most observability tasks. STDOUT (logging) usually went to a terminal via a serial port (creating all sorts of issues with blocking that at least did teach me a lot about asynchronous i/o).

Today there are a number of patterns for centralized log delivery that work in virtually any language and operating environment. Entire libraries exist for event telemetry - really just a way of sending specialized log messages that provide specialized information about what is happening or what part of the code happens to be running.

Unfortunately, I see all too many systems and applications that do not implement observability. As mentioned in a recent post( https://t.co/sCG0zi42GK ), I grew up on the principle of Measure, Monitor, Manage. Paraphrasing a former mentor and boss from many years ago, "You have to measure something in order to monitor it. You have to monitor something in order to manage it. You have to manage something if you want it to work all the time."

Scalability requires observability. If you cannot see what is happening and understand the traffic your applications are getting, you cannot identify and fix bottlenecks. If you don't know how long it takes to process requests across a wide range of traffic scenarios, you won't know how your application will perform under stress.

Ask your team members responsible for level 3 support, provisioning and performance tuning where their blind spots are. Chances are you will identify some quick wins - areas where you can add event telemetry feeds (simpler as first step than refactoring logging) or implement automated pattern searching on centralized logs you already have. In terms of tech debt resolution, this effort is one of the highest value, quickest return areas out there!

[Dave Thompson @3leapsdave]

Remembering that Excellence Wins

Here's a plug for a book that reminds me that customer satisfaction and (fanatical) loyalty come from a firm's absolute devotion to operational excellence, empathy, kindness, and the incredible power of being nice.

Ever have a book come to mind when writing? Only a fistful of business books stick with me. Today I was working on a Smooth Development Maxims post for our 3 LEAPS page and one came to mind. The subtitle for this post is "Measure, Monitor, Manage" - a core maxim taught to me by a gifted boss and mentor back in the late 80s and early 90s.

Check out the book, Excellence Wins (https://t.co/fVtAkBjJc1) by @horstschulze . Mr. Schulze was a co-founder and longtime president of Ritz-Carlton and the architect of their legendary management systems. This book will remind you of how to build lasting customer loyalty. I offer that it might inspire you to be passionate about building a better organization, being a better leader, and being a better person. Without spoilers, I will simply say that his idea of "becoming the best in a world of compromise" is something I strive to follow.

Ritz-Carlton was legendary for their service in the 90s. Their employees were (and may be still) famously empowered to spend - without management approval - a fairly large sum "per incident" in order to delight a guest. What brought me into researching them was their process discipline. As my mentor taught me, "You have to measure something in order to monitor it. You have to monitor something in order to manage it. You have to manage something if you want it to work all the time." Does it seem strange that I learned a core principle of enterprise-scale operations from the business processes of a hotelier?

The funny thing about this (important-but-dry) topic is that I learned in the context of how a hotel chain became literally world-class in service delivery. Those of a certain age may remember that Ritz-Carlton was (and still is) the only hotel company to have won the Malcolm Baldridge National Quality Award - twice (in 1992 and 1999). Here's a link to a story about win #2: https://t.co/uPTPQATtc1.

If you deliver solutions with software, I humbly note that "people buy solutions from people." Those feature charts we make help us stand out in the early sale. Winning and keeping clients comes from delivery excellence. In the end, I've come to appreciate the brilliance of my former boss and mentor, teaching me that software folks could learn quite a lot from the hospitality industry. I try now to seek out examples of firms delivering excellence in any area or walk of life. People reward firms that deliver good and reliable experiences with more business. People become lifelong fans of firms that deliver excellence and have genuine empathy and initiative for their needs and for those around them.

Don't compromise on principles - deliver excellence!

#opsuccess #smoothingsolutiondelivery #3leaps

[3 LEAPS @3LEAPS]

Smooth Development Maxims - Keeping Things Dry

One of our core maxims is to "follow the DRY principle." DRY means "Do not Repeat Yourself." It means that - among other things - system-of-record data should not be duplicated. Advanced data engineering and analytics staff will note that we DO repeat ourselves in certain cases, particularly with semantic data stores built from flattened data into "query friendly" layers. I stand by the DRY principle even here because of how these flat data stores are typically built. Whether through views ("simulated" data stores built on lower layers in a user-friendly way) or via automation, the DRY principle ensures we have only one system-of-record.

The web developers among you may use pnpm, an alternative to npm for managing the (many) packages in a typical web application. One of the key innovations is the use of a content-addressable store. In straightforward terms, pnpm follows the DRY principle. Have 100 modules using the same package? You'll have one version shared across the 100 modules on your build machine rather than 100 copies.

Staying DRY is not only about saving disk space by eliminating duplication. Being clear about where the "source of truth" lies leads to better design. When we make copies of the same thing over and over, it is highly likely that at some point we'll mess up an update and change only SOME of those copies.

Devops professionals probably use Terraform or OpenTofu. These tools (used for efficient provisioning of infrastructure) use a language that can in some cases require lots of copies of the same text. On this front, 3 LEAPS supports Gruntwork's open source work on Github. In particular, we are fans of the package Terragrunt (please consider supporting if you are building IaC). We like this tool for many reasons, particularly as it helps us to avoid duplicating common code.

What are some things we can do to stay DRY?

1.) Do input validation in exactly one place (using schemas - for a future post)
2.) Put configuration data in exactly one place
3.) Structure code as a series of transformations and extensions that derive additional measures from existing sources (vs. copying functions and altering)
4.) Build specifications referentially (don't repeat requirements or parameters - instead reference other sources)

Despite the simplicity of this principle, we see a lot of (ill advised) duplication. Imagine a case where you have three slightly different paths in a pipeline using the same input. If you copy the code twice, now you must remember to make common updates multiple times. If you forget, some cases will work and some will fail. As obvious as this seems, I routinely coach development teams through bugs that arise from improperly duplicated "common code."

DM me if you would like some references on this topic. Take a look at your work in-flight and ask yourself where you might be copying rather than referencing.

Keep your systems DRY!

#smoothdevelopment #staydry #3leaps

[3 LEAPS @3LEAPS]

Smooth Development Maxims - Making Specifications Work

Make specifications sparse, precise, and verifiable.

In the TL;DR age, too much documentation is unfortunately nearly as much an issue as too little. Modularity is important - just as it is in software design. Specifications represent the most important part of any system's documentation tree. Good specifications provide developers, architects, test analysts and other product-related specialists with clear requirements and methods for verification in a minimalist structure. We describe this as "sparse, precise and verifiable.

The most effective specifications are built from many single-topic items. While complex components or interfaces may benefit from an additional "theory of operation" in prose, focus first on quantitative requirements that map to success. Build a taxonomy of the important kinds of specifications. Then build small (Markdown-based) documents for components, interfaces and systems that cover a single topic. We can use auto-build systems to build a larger document by combining many small ones.

Working in this manner makes it easy to get started and to focus on the areas your particular system needs most. In our @3LEAPS Flow Lanes syntax, we always include what we call Key Assurance Metrics: testable attributes tagged as functional, performance, availability, and security. These key inputs work to an overall verification plan, representing the KPIs that define success.

At the component or interface level, start by deciding what needs a specification. Interfaces should be precise - JSON Schema, JSON Type Definition, OpenAPI, and AsyncAPI are tools you might consider. External interfaces in particular need formal specifications that are accessible by (external) users.

Document algorithms and functional logic using simple flow diagrams or text and by storing the (Markdown) documents in the code as noted in another post ( https://t.co/uwFLpudCKy ). When reviewing what constitutes an adequate specification, think from three key perspectives.

1.) What does a developer need to know in order to address the full range of inputs and boundary conditions? Consider numeric precision as an often overlooked item.

2.) How are high-level system requirements (Key Assurance Metrics) particularly impacted by this component?

3.) How can a test analyst confirm operation in black box fashion?

While the specification count might seem overwhelming, recognize that no system and no component requires every type of document for every use case. Start with new work, adding specs that cover the most important areas based on your own analysis of where gaps lie. Ensure the acceptance process for each development cycle includes specifications.

As you build up an inventory of specifications, your devops team can add CICD operations to create a larger validation "suite." Don't let TL;DR attitudes create ambiguity or inconsistency in your system design!

#smoothdevelopment #flowlanes #3leaps

[3 LEAPS @3LEAPS]

Smooth Development Maxims - Keep Documentation Close

Keep Documentation Close to Code

The second of our Smooth Development Maxims focuses on the accuracy, timeliness and relevance of documentation. In the days of waterfall-style software development, we used to write requirements docs, high level design docs and then detailed design docs. Assuming all these were actually completed, then people would start writing code. Guess how relevant the documents were after a few months?

Agile processes do not automatically solve the documentation problem. It takes discipline from all team members and leaders to keep the documentation accurate as designs and interfaces iterate. Noting the expression "out of sight, out of mind," the best place to keep technical documentation is in the same repository as the code (or one nearby.)

Here are some success attributes that indicate a good documentation process:

1.) A README exists in each repo with accurate crosslinks to other docs - Is this a true "start here" with an accurate overview? Do the crosslinks to other docs work?

2.) Include comprehensive and accurate setup instructions - do they work? Can someone new to the project follow the (markdown) content and get the code built?

3.) Include links to styles and standards guides - since these generally apply to multiple projects, update centrally. Make sure links work. More to come on styles and standards in a future post.

4.) (Short) docs for component-specific topics such as context diagrams (i.e., C4 syntax), theory-of-operation summaries, or specialized integration notes are present and getting updated

5.) (Relevant) specifications included "in repo" - review and update as needed

6.) One or more auto-build scripts are running to build schema docs, interface specs and supporting materials for user guides - steady state one should never build docs for distribution manually

Larger docs for the enterprise such as runbooks (devops), roadmaps and agile artifacts (prod mgmt), (external) integration guides (dev + prod mktg) should generally be assembled from smaller pieces. Build managers and scrum leaders need to ensure the checkpoint processes at each milestone include review of the documentation elements relevant to the changes.

Attention to detail matters. If your documentation is not working for you, start a process to add those small elements mentioned here in the next cycle. Once everyone gets comfortable with the incremental changes, you can retrofit the larger build and integration elements over a few cycles. With complex systems, both internal and external users benefit from accurate documentation. Boring though it may be, getting these small bits right is the best way to ensure everyone has a big picture that is visible and accurate.

#smoothdevelopment #embracingcomplexity #flowlanes #3leaps

[3 LEAPS @3LEAPS]

Use Plain Text for Documentation

In the 3 LEAPS Flow Lanes development system
( https://t.co/9in96NoNQf ) the statement, "Use Plain Text for Documentation" is our foundational maxim for development. Everyone says documentation is important but almost no one does a good job of maintaining it. Complex systems have different documentation types and audiences, ranging from interface specs and schemas to user-facing manuals. How you render and present such information varies. How you build and archive these types should be mostly the same.

Here are the key attributes we have for a plain text documentation approach:

1.) Text-based file formats - don't use file types that you cannot view in a text editor

2.) Human-readable content - while SVGs are text-based, no one can "see" a diagram like in the Matrix

3.) Changes viewable using only simple "diff" tools - readers must be able to understand what changed by comparing the source files in raw form

4.) Markdown-based structure for text - pick a standard flavor (DM @3leapsdave ) if you need suggestions) and use it literally for everything (specialized extensions allow use of TeX and LaTex for rendering equations, as example)

5.) Auto-generated diagrams wherever possible - we recommend @MermaidChart for all but the largest "big picture" diagrams where manual layout matters

6.) Source-control stores actual source - PDF is not a source form unless it is the authoritative reference from a third party where no human-readable alternative exists

7.) Limits use of SVGs and avoids binary images as source - apply #5 and #6 except where an image is "the" source for a crucial part of a document

8.) Packaged content auto-built using scripts or CI/CD - use Pandoc, MkDocs or something similar to blend content and transform Markdown into the final published form such as HTML or PDF

Applying this first maxim is a good way to lay a foundation for usable, relevant documentation. Just as we do not build code by hand in a real system, one should not be building or transforming documentation manually either. We'll explore some other documentation-related maxims in future posts.

You cannot scale smoothly without good documentation. Commit to a process built on plain text and human readable content as an important step towards better documents and a better system!

#smoothdevelopment #embracingcomplexity #flowlanes #3leaps

[3 LEAPS @3LEAPS]

Invent as a last resort: Integrate First

We have a saying in enterprise product management that "we get there faster with many in the boat." Basically, this means that complex functionality almost always gets developed faster and works better when we collaborate with multiple customers having similar goals. Shaping planned work as we adapt to different clients' needs forces product managers to "horse trade" - smoothing edge cases and blending needs into more generalized forms. Innovation for one case can extend to others.

This sentiment does not always carry over into enterprise architecture. Though we often say things like "we stand on the shoulders of giants," using the ancient metaphor that goes (much) further back than Isaac Newton, we often miss the mark. Too often we see well-intentioned teams rebuild an existing building block or component from scratch, unaware of the existence of working components or emphatic about the need to recreate due to edge cases.

At 3 LEAPS, our Smoothing Solution Delivery process is built around a strategy quadrant that helps organizations align the approach and activities of different teams. On the horizontal axis, we span sourcing strategy from "Develop First" (on left) to "Integrate First" (on right). Excluding an extremely narrow class of applications, almost everyone should think "Integrate First" most of the time. Reserve the "Develop First" approach for components ("wheels") that must truly break new ground in functionality or scale.

Let's consider some practical tips from this Integrate First maxim that apply to most (data-intensive) applications and systems today.

1.) Don't design components in the subfloors of your applications

Messaging, client-side frameworks, AAA, encryption, telemetry, queues, common file parsers, and web servers are all examples we've seen recreated unnecessarily from scratch in larger applications. Unless these components ARE your product, you (almost certainly) should not be designing them. Integrate first!

2.) Don't recreate ETL / ELT

Less time writing loader code means more time doing useful things with your data. Someone else already solved this problem for you. Let folks who think about these problems every day innovate here!

3.) Look for "big tent" components

Broad and diverse usage improves stability. The more real use cases a common component has in the real world, the better it will be. Leverage the experience (and bug fixes) of others!

4.) Strong communities (probably) are a sign of success

Popularity (i.e. Github stars) should not drive component selection. Often, It is an indicator of what does matter: references, growing user community, frequent & valuable updates, friendly support, and good documentation.

All other things being equal, your solution and your enterprise will scale more smoothly if you mostly integrate proven components using sound patterns. Invent sparingly and only as a last resort!

#smootharchitecture #integratefirst #enterprisearchitecture #3leaps

[3 LEAPS @3LEAPS]

Smooth Development Maxims - Embracing Complexity

At 3 LEAPS, we shape software development and delivery activities for ourselves and our clients around a loosely grouped set of structures and principles that we call Flow Lanes. Complex, data-driven systems - regardless of language or application - share some common themes. Handling them well is what we call Embracing Complexity.

1.) Highly transactional operation and analytics

* Units of work (commercial / retail transactions, process step updates, location updates, etc.) are numerous and occur irregularly, often in bursts around peak periods or external events

* Transactions reference a large set of supporting data in multiple dimensions, representing actors, items, locations, and other descriptive attributes, often structured in hierarchies that may change over time

* Operational systems have specific response time requirements that are usually "human scale", with little tolerance for extended completion during busy periods

2.) Highly volatile data stores

* "Hot data" generally means (expensive) hot storage needed

* High write rates exacerbate issues with locking and consistency

* Large data sets

3.) Highly variable load

* Reactive loads driven by user actions may create cascading "ripple" loads

* Scheduled and background batch-style actions may interfere with user performance

4.) Possibly vast data scope

* Frequent need for various rollups and period summaries that require on-the-fly creation or data-driven updates due to moving time windows and changing filter criteria

* Common "summary / detail" patterns that reference specific transaction and underlying members separately

While there are still reasons why we might separate operational systems-of-record from analytical data stores, advances in data store technology, storage performance, and middleware solutions mean the lines are often blurred. We typically architect systems like these around some common principles:

* Use of event-driven data pipelines with support for multiple subscribers and branching workflows that evolve as new applications or transformations are added (scale additional logic smoothly)

* Design for idempotency - a fancy term that means re-running a step multiple times will always yield the same results (handle retries cleanly)

* Deliberate data consistency, meaning that components and subsystems are aware of how updates are propagated across the system (ensure consistent results)

* Deep instrumentation for observability, which is a fancy way of saying there are a lot of telemetry signals and progress updates that are collected for use in management tools (provide clear status in drill-able form)

In this new series on Smooth Development Maxims, we will explore strategies, tools and patterns that - in our experience - ensure reliable, scalable operation of these highly transactional systems.

#productarchitecture #smoothdevelopment #embracingcomplexity #3leaps

[3 LEAPS @3LEAPS]

Maxims for SaaS Success - Infrastructure and Tenancy Infosec

Continuing from part 5 ( https://t.co/BGenuBNV35 )on the infosec front, we explore two areas in SaaS operations often lacking attention. Proper (external) monitoring provides both the critical early warning signals for incidents and the baseline forensics needed for detection and containment. Proper system and data isolation help prevent catastrophic cross-tenant or cross-tier breaches that create information leakage or compromised results for clients.

We often see these areas ranked “#5 on our top 3 list” - important but not reaching the threshold for action. Talk with your CIO, CSO and architect leaders to see what needs doing. Use risk and impact assessments to prioritize investments. Make incremental progress where you can, rather than waiting for next year’s budget. Perimeter security is not enough!

1. APM (Availability and Performance Monitoring + Security and Information Event Monitoring)

We still see a lot of SaaS applications that have no externalized, automated monitoring. There are many bespoke solutions and specialist MSPs that handle these tasks thoroughly and cost-effectively. DM me if you need some suggestions! Operational success starts with knowing the state of your applications and session traffic. Don’t put blinders on your infosec team!

2. Proper tenancy and tier isolation controls

While most recognize the importance of separating production and test data, we still see incomplete or inconsistent isolation of customer data and assets. For multi-tenant architectures, what specific controls exist to minimize the risk of improper client data mixing? What “four eyes” (two person) processes exist for operations that change such controls (to protect against malicious insiders operating undiscovered)?

Tier isolation can be extremely complex in optimization systems where models and certain core data may be used simultaneously by “staging” and “prod” environments. Multiple control schemes are needed to prevent accidental pollution of tier data - use role-based IP access and different credentials to force a conscious review of each connection and deployment action.

Use firewall controls, tier-specific SSH keys (with different passphrases!), different DB access credentials, etc. to ensure operators do not accidentally “talk to the wrong system.” Ensure application access credentials are separate by tier with no exceptions. Saying sorry won’t make a client feel better after a cross-tenant data breach or tier pollution case!

We often think about perimeter security with a mental image of masked hackers who say “I’m in” while using ASCII terminals in dimly lit rooms. External firewalls are important, but real security requires more than a high fence. Invest in observability, automated monitoring, self-service session management, and multi-layer asset isolation. These items are more than $5 words - your reputation and your client’s information security depend on them!

#saasmaxims #productcontinuity #infosec #3leaps

[3 LEAPS @3LEAPS]

Maxims for SaaS Success - Product Additions for Infosec

The origin and growth vectors for a SaaS firm often set the tone for security. Those firms handling extremely sensitive information know that compliance hurdles must be met even for pre-launch testing. Experienced founders or institutional investment often prioritize security processes and compliance topics sooner based on playbooks or past experience. Much of the foundational work focuses on access controls, perimeter security and process consistency.

At 3 LEAPS, we often see some areas “in the product” that are often overlooked regardless of origin vector. Continuing from part 4 ( https://t.co/TNqNRSmwGc ) in our series, here are two of the most important.

1. SSO and MFA - available, standard and self-service

Setup processes for MFA and SSO (and any user provisioning) must be automated - full stop. We’ve written previously how SSO should be a standard feature where a client has 5+ users. The support for one or both is often relegated to a vague “enterprise tier”, which may keep the firm from implementing a “productized” setup and maintenance scheme. A well-crafted SSO implementation should be self-service for client admins, requiring fewer than 30 minutes to set up. Every major web framework today includes well-supported libraries for MFA. Help your clients keep access controls tight as users come and go!

2. Centralized session management - visibility to and control of active sessions

What happens if someone has a password compromised? Know someone who has had fraud related to a financial account? Imagine how maddening it would be to see an attacker “stay in” a system even after you changed your passwords. What if you didn’t even know if someone was accessing your account? It is security “table stakes” to maintain both an audit trail of sessions and an active, real-time sessions table. Proper handling here requires thoughtful design. Here are some of our “in product” recommendations:

a. Users can terminate active sessions from inside the app

b. Users can review session history in app, with timestamps and IP addresses

c. Solution has account-and-client-specific circuit breakers that block new logins AND terminate existing sessions, accessible by SaaS admins and automated systems

d. Password changes require logout of current session and re-validation with MFA (when SSO not used)

Get some help from an experienced security consultant to be sure your processes are appropriate for your situation, particularly if your system deals with sensitive PII or acts as a system-of-record for MNPI (Material Non-Public Information). Work with a trusted external consultant or MSP to test for weaknesses using real-world attack vectors. Findings can be prioritized for resolution by risk and impact. Some relatively simple additions to your web applications and operations console can reduce your vulnerabilities. Using SSO helps but is not enough!

#saasmaxims #productcontinuity #infosec #3leaps

[3 LEAPS @3LEAPS]

Maxims for SaaS Success - Operational Success

Operational Success is about ensuring clients and customers can do what they expect with your solution, when they expect to do it. See https://t.co/23PfaYMNRx for part 3 of the series on Product Strategy. Here are some maxims common to complex solutions that drive operational success.

1. Instrument (literally) everything in your solution.

Don’t forget data pipelines! Observability is table stakes in today’s complex distributed architectures. Use industry standard tools, integrating in your own hooks and feeds. Use code as the first line of detection, sending anomaly cases to reviewers as warranted. Refine and extend instrumentation in each new release!

2. Automate everything in the core build->deployment->test paths.

Don’t forget rollback paths! Adapt the deployment model (big bang, canary, blue/green, etc.) that aligns to your scale and update frequency. Manual deployments are for early development and emergencies only.

3. Be transparent on availability.

Use status pages, ticket system integration, Slack channels - whatever works for you and your clients. Make any calculations for SLA self-service. Be sure to host availability tracking and reporting apart from your that of your other systems! Transparency builds trust and accountability.

4. Recognize that accountability is the price for stickiness.

SLAs matter, particularly for enterprise SaaS. Be precise about methods for issue identification, detection and resolution. Agree up front on measurement criteria and process for communicating “after action” activities.

5. Train, drill, test and review regularly, in the (real) systems where it matters.

Design and run simulations that are as close to real life as possible. Encourage a “no finger-pointing” culture to make each run better than the last. Kaizen!

Enthusiastic outreach and a great value proposition drive the (initial) interest in your solution. Trust in your operations drive retention and referrals.

Do you have policies or practices that have worked for you in the past? If so, please comment below.

[3 LEAPS @3LEAPS]

Part 2 of our series on Effective Incident Response, focusing steps that organization leaders can take

[Dave Thompson @3leapsdave]

7 Steps for Effective Incident Response - Organization Leaders

When something goes (very) wrong with a solution, it is vital that teams and leaders keep calm and remain grounded in the present. "In crisis" is no time for "why / who" recriminations! Organization leaders set the tone through culture, leadership style and support.

Whatever got you to this point matters (during the incident) only to aid in diagnosis, containment, and resolution. In part 1 ( https://t.co/S2nfAtlqY5 ), we focused on recommendations for the IT First Responders. Here in part 2, we continue with recommendations for the organizational leaders. We’ve refined these through both our own experiences and through work with @3LEAPS clients.

1. Protect the time and concentration of IT First Responders

Designate an "Action Coordinator" who understands the product and customer base to rally other and to protect the responders.

2. Designate and empower an “External Communication Leader” to own the messaging activities

Best practice for incident response is to have a leader designated ahead of time who is the “goto” SPOC for external messaging, supported by (other) senior executives as needed. Review, refine and approve requested updates promptly!

3. Lead on a “Response Clock”

Don’t overwhelm team with ad-hoc “status update” requests! Instead, structure a periodic, regular “response clock” where the update cadence varies (hour / day / week) as incident progresses. We implement both high-level (“updates”) and detail (“action”) channels in the collab tools so people can follow selectively.

4. Support those on front line of response

Rarely do we resolve issues during normal business hours. Remind managers to give breaks and work out time off after the incident. Remember their families also during long-running issues!

5. Discipline in private (wherever possible, after the incident is resolved) and praise in (enterprise) public

Take time to thank those who responded and give them a chance to blow off steam as needed. Handle disciplinary matters behind closed doors, following organization policies.

6. Prioritize improvement through "post event actions"

Focus discussions on root cause and improvements, without finger-pointing. Use the opportunity to demonstrate commitment to learning and improving. Seek out specific plans for observability and automation investments that will have broad impact.

7. Require (and support) organization to improve with simulations and drills

Outages and service interruptions happen to everyone - literally everyone. Like sports teams and music ensembles, it takes both individual and team practice to become proficient at incident response. Past certifications and accolades do not grant immunity!

Keep your teams sharp through drills and "after action" programs. Make sure your IT first responders feel both a sense of ownership “in” and a sense of support “from” the organization. Stay ready to respond effectively!

#saas #incidentresponse #productcontinuity #3leaps

[3 LEAPS @3LEAPS]

Part 2 on Making Personalization Work

[Dave Thompson @3leapsdave]

Making Personalization Work Part 2 - Time to Plow the Ground

"You can talk about the farm or you can plow the ground" - Chad Brock

If I could make a theme song for personalization transformations, it would be Lightning Does the Work (listen on Spotify at https://t.co/LR4YcyJJvL) by Chad Brock. 

Our instinct is often to say "let's get the data first". I feel this too, having operational responsibilities myself for many years. But, the markets wait for no one!

Since there aren't any country songs about icebergs (see part 1 at https://t.co/yMvKIfTudk), let's view personalization scaling as setting out two crops - one for an early harvest of "quick wins" (campaigns to core contacts) and one for a larger late harvest (outreach scaled with our personalization transformation).

Here are some recommendations formed from many optimization projects.

1. Choose and adapt transformation scale wisely
Everyone starts from a different place on the curve. Are you revitalizing a couple of houses, a street or a citywide waterfront area in your personalization journey? Adapt your approach to the scale of your goals!

2. Commit to continuous data quality scorecarding
Understanding your current data quality is first priority. Begin outreach activities using your high-quality core contacts (customers, prospects and engaged followers). Scale your processes and tools while you are conducting those first (small) campaigns!

3. Build around a solid CDP core
If you have a B2C or a horizontal B2B audience, you need a CDP. The right solution depends on your scale and existing platforms. I really like the approach @RudderStack takes to leverage your existing data warehouse. Multiply the benefits of personalization investments with a solid CDP core!

4. Build continuous learning into your campaigns
Instrument workflows around every outreach error (bounce backs, etc.) so every mistake or miss generates a corrective task automatically. Firms such as @Optimizely can help you scale A/B testing and improve targeting. Learn from every campaign!

5. Scale personalization by embracing automation
Personalization makes the content adaptation process EXPLODE! Consider firms like @HunchAds to partner with you in automating your entire outreach process, down to rich content repurposing. Free up your creative staff to do more creating!

Personalization is key to being seen and heard today. You need tools to do this properly. Get help from a consultant (my plug for https://t.co/ox1IGmFMF3 😉) if you need a hand assessing data quality or building a transformation plan that balances early wins with long-term growth.

Build (better) your MarTech stack and start plowing those fields!

#martech #productmarketing #personalization

[3 LEAPS @3LEAPS]

Maxims for SaaS Success - Product Strategy

1. You want a community, so build an ecosystem
Prioritize interconnections through APIs and integrations. Make data portable. Support external developers and integrators as a primary "stakeholder group" for your products (with approach refined to your strategy).

Focus on connecting with the tools and data sources your clients' need most. Communities arise most quickly around "federated" solutions that reduce task attention surfaces, automate large processes (usually across multiple tools), and generate unique insights for users. 

Think using that old adage that "1+1=3". The whole of a (solution) network is greater than the sum of the parts. A powerful community creates a center of gravity that pulls others in to create more opportunities!

2. Make value measurement self-service
Represent the voices of your clients and your client partners by building in value measurement tools and creating quality content to help them increase the value even more. Avoid the rush of "value justification" at contract end.

When your client has value measurement right at hand using agreed methods and assumptions, you spend less time on contract renewal and more on helping clients grow!

3. Shape and prioritize "big tent" enhancements first
Good product managers capture and prioritize requirements by sizing opportunity and need.

Great product managers blend and shape requirements through dialog with clients to amplify the impact of each product investment wave.

The most valuable enhancements are those that address many use cases. Be open with clients about need to combine requirements and shape for the intersections first. They will help - everyone wants someone else in the (new capability) boat!

#saas #saasmaxims #productstrategy

See our @3LEAPS feed for more "Maxims for SaaS Success" topics. We'd love to hear your thoughts!

[3 LEAPS @3LEAPS]

Maxims for SaaS Success - Core Principles

1. First impressions drive the brand
Clarity in offer, service and transparency in operations all build the trust that drives retention and referrals!

2. Passionate users must first be proficient users
Listen, support and teach across their preferred contact channels!

3. Simplify the success of your clients
Choose between BEING or CATALYZING your clients' SPOG (Single Pane of Glass) for important process(es).

Success usually involves less manual work and fewer apps + screens.

Either can work. Trying to be both confuses people.

#saas #saasmaxims #productbranding