Experiment to prevent rigor corpus - part 4
I describe the documentation needed to facilitate a good experiment. This is the final post in this series.
This is a series of four related posts:
- part one introduces the experiment
- part two shows 5 steps to implement
- part three discusses challenges
- part four illustrates the documentation
Documentation is a necessary evil. Writing forces us to clarify our thinking in a way we cannot in our heads. But sometimes writing turns into busy work or worse, fancy-business-writing. The best writing strikes the right balance for your needs and those of your reader. Be direct with what you are saying and use the least amount of words possible to convey your meaning. Yes, writing less words takes longer.
In this case the writing forces a bit of scientific rigor into the process. Not only will this help clarify thinking, but will put leadership more at ease. The following questions make for an easy to use protocol:
Define the problem you are trying to solve
Describe the change you are making
How long will this experiment last
How should this change fix things
Expected side effects of the experiment
How will we measure the effect
How often will we make sure it is working
Implementation plan
You are writing to clarify your thinking and gain a complete common understanding. Deviate from the order if you must, but I recommend completing them in the given order.
Define the problem you are trying to solve
Clarify the specific problem you want to fix. It should be specific and measurable. This both gives a focus and establishes scope. This prevents the natural tendency to solve other problems at the same time. Everything in this experiment should work toward solving this problem.
Bad: “Fix helpdesk process”
Good: " Reduce the time between reporting and triaging a ticket"
How will we measure the effect
You need to articulate how you will measure success. This measure should align with the problem statement. Remember we need data, not opinions. Measurements can be very tricky so spend some time on ensuring clarity.
Bad: “time it takes to triage the ticket”
Good: “The number of working hours reported in weekly status report shown for time-to-triage will be lower.”
Describe the change the experiment makes
This clarifies the decision the team made. The implementation details will come later, but for now add enough detail, so the change is clear to everyone.
Bad: “tickets will be triaged sooner than before.”
Good: “The staff will set reminders to check the inbox every 30 minutes.”
How the change will fix things
Put down in writing the the effect the change will have. This is very helpful a month later if the change is not working. You can look back and recall what the thinking was. When coupled with the change description it paints a clear picture of intent.
Bad: “Tickets will be triaged faster.”
Good: “The 30-minute check by staff will result in tickets being seen sooner than the current daily check.”
Expected side effects of the experiment
Process changes can have downstream impacts. Considering any impacts can help you communicate then beforehand. Side effects surprising people will not do anything to support your experimentation!
Bad: “Staff will have to check tickets”
Good: “There will be a minimal impact on staff performance as they will be triaging more often.”
How long will this experiment last
You do not want a lot of experiments languishing in production. Deciding on an specific duration / end-date you establish a frame for the experiment. You can schedule decommissioning or expansion for that day ahead of time.
Bad: “A few months”
Good: “It ends January 10, 2022”
When will we check on the experiment
You must keep an eye on your own personal Frankenstein’s monster in case it runs amok. There may be unintended consequences you will want to catch early. This question sets up a regular cadence to review the measures and make adjustments. At these points you may decide to modify, cancel, or permanently implement the change.
Bad: “Frequently.”
Good: “At the weekly staff meeting”
Implementation plan
The implementation plan can vary from simple to complicated. At the least it will cover what needs to be changed, who will change it, and who it needs to be communicated. In some cases this is only a few lines. In others, it could be a full project plan. I suggest balancing the level of detail with the complexity of the change or risk.
Future looking questions
Ester proposes a few forward-looking questions. These are not unreasonable, but do not need be decided at the beginning. Thinking about next steps has value, but can considered at a follow up meeting.
If it is not working we will…
If it is working we will…
Both questions lay out future scenarios by setting some expectations around the next steps in the process. They can be as simple as “We will implement fully” and “Revert back to the normal process”. They may also have more nuance such as “If it is working we will reduce the check time to every 15 minutes and re-evaluate”.
Does it work?
From my experience I would say the results of these broke down into:
• 25% worked well enough to keep it as is as a permanent replacement.
• 50% worked somewhat, but the lessons learned directly informed the real process and enabled controlled improvements or follow up experiments to build on the success.
• 25% turned out to be not workable, but we learned something from them.
Additional reading
Change Artist Super Power: Experimentation (estherderby.com)
