WOPR22 Brainstorm: Illusion of Reality

Our last brainstorming exercise was an examination of the ways in which we acknowledge our performance model is likely to be inaccurate, compared to reality. Since much of performance testing centers on “realistic” simulation, and iterative development forces us to test faster with less information, it was instructive to review how we approach simulation fidelity.

Substantial discussions took place under each of these points. These are a “Table of Contents”, at best.

One of these begs some explanation: “Turkey Theory”. Essentially, this refers to the fallacy of assuming that the past completely predicts the future. The example here is the Thanksgiving Turkey, who has the experience of several hundred days to support the proposition that the farmer loves him, feeds him well, and wants him to be healthy and happy. On one particular day, the prediction that today will be the same as yesterday becomes tragically (and deliciously) wrong.

This work is the product of all of the Workshop on Performance and Reliability (WOPR22) attendees. WOPR22 was held May 21-23, 2014, in Malmö, Sweden, on the topic of “Early Performance Testing”. Participants in the workshop included Fredrik Fristedt, Andy Hohenner, Paul Holland, Martin Hynie, Emil Johansson, Maria Kedemo, John Meza, Eric Proegler, Bob Sklar, Paul Stapleton, Andy Still, Neil Taitt, and Mais Tawfik Ashkar.

Load ModelsEnvironmentReporting and Projection
Single UserDifferent/Less HardwareMaintenance of Production Environment
User CountsNon-dedicated Test EnvironmentOver-Simplification
Mix of WorkflowsStubbing/ignoring backend and other systemsGenerating more reports under the illusion that existing reports are consumed
Pacing of user Input & DelaysBackground processing and maintenance are not consideredMisuse of statistical analysis
Optimized Data - row counts and cardinalityInfluence of external systemsShifting acceptance criteria
Very different test dataImpact of marketing/social marketing (point load)"Turkey Theory"
Missing incomplete sessionsShared infrastructure resources
Not evolving load model and/or dataVariable network models (consider Shunra and LANforge to address)
Organic loads and arrival ratesWarmed Caches
Tool limitationsInconsistent/dynamically changing configuration, software, and hardware (cloud and/or virtualization)
Tool limitations
Updated: June 4, 2014 — 10:00 pm

Leave a Reply

Your email address will not be published. Required fields are marked *

© 2003-2017 WOPR Frontier Theme