Objectives:

 The main purpose of the WOPR sessions is to build skills in system performance and reliability, and allow people who are seriously interested in these topics to network with their peers.

Another purpose is to help build a community of professionals with common interests. To this end, the website contains a growing repository of WOPR papers that were written as a result of previous workshops. TOP

Focus:

 WOPR’s primary focus is on evaluating system performance and reliability. This includes performance measurement, load and stress testing, scalability testing, reliability measurement and evaluation, and system and product certification.

Important but secondary areas of interest are broad, including:

Performance and reliability engineering: system design, network design, database design, self-tuning systems, and self-healing systems.

System functional testing: test automation frameworks, quality metrics, test planning and test case design, test case and test data management, data analysis and test results evaluation, test environment management, test tool development, test tool use, design for testability, software fault injection, failure analysis, building defect databases, and test lab versus live field testing.

Operational management: capacity planning, rightsizing, disaster recovery planning, system administration, database administration, network administration, performance tuning, bottleneck identification, troubleshooting, debugging, resource utilization, performance prediction, reliability prediction, cost/benefit analysis and justification, service level agreements (SLAs) and quality of service (QoS), and managing vendor relationships.

Management: managing teams who work in the areas above.

WOPR is neither vendor-centric nor end-user-centric, but accommodates a mix of viewpoints and experiences. The area of interest is not limited to high-availability systems, real-time systems or life-critical systems. The range of potential interest is broad, from single-device embedded systems to complex ecosystems which are multi-tier, multi-technology and multi-vendor, and which support multiple interacting applications and databases. TOP

Membership:

 Our constituency is people who are interested in system performance, reliability, testing and quality assurance. Membership is open to all. There is no charge for membership at this time, though membership fees may be introduced in the future to fund WOPR activities. To join, register via the member signup. Members receive occasional newsletters on system performance and reliability. Members can elect to dissociate themselves from WOPR at any time. TOP

Style:

 The heart of each workshop session is a series of experiential presentations and group discussions, which focus on the chosen topic of the particular workshop session. The atmosphere is collaborative, supportive and constructively critical. Each person typically presents his or her findings from a recent and topical project. In other words, we sit around telling whoppers (WOPRs). 

WOPR sessions engage their participants much more than traditional conferences and seminars. A typical conference allows lets say 35 minutes to present a paper followed by 5 to 10 minutes for audience questions, which means there is little interaction and cross-examination of the ideas. WOPR proportions are often the opposite: a 35 minute presentation may be followed by two hours of discussion and detailed critique if the group finds the information exchange fruitful. Some conventional conferences are marred by fiction and fantasy which are presented as facts, sales pitches and self-aggrandizement. At WOPR, the friendly but close questioning tends to eliminate these. TOP

Selection of Participants:

We target the attendance for a workshop session at 10 to 20 people. Participants are selected in part to give a well-rounded set of experiences and perspectives. If a particular workshop session is over-subscribed, we will be forced to turn away some qualified people – seats are not guaranteed. A limited number of beginners are welcome in a session, and we accept them based on their expression of enthusiasm and serious interest in the topic under discussion. Our goal is to finalize the list of participants for each workshop and notify each participant at least six weeks before the first day of that workshop. TOP

Double-Track Sessions:

 If demand is sufficiently high (35 or more people), we will split a WOPR session in two and conduct simultaneous tracks on different topics in adjacent meeting rooms. People will be free to change between tracks, within reason. Social activities such as refreshment breaks will be done jointly. TOP

Themes:

 We select a topical theme to provide a specific focus to each workshop session. Examples of possible themes (one per workshop session) are: (a) factors influencing the success of performance, load, stress and robustness test projects, (b) strategy, approaches and testing techniques for particular types of projects, (c) limitations and common misuses of load test tools, (d) performance data analysis and interpretation, (e) scalability testing, (f) setting up and using the load test lab effectively, (g) the relationships among testing, tuning and debugging, and (h) testing the 3Rs (reliability, robustness and resilience). There are dozens of other promising candidates for the theme of a workshop session. The organizers will participate and vote on the selection of the topic for the workshop session.

TOP

Presentations:

 We encourage each person attending a session to give a presentation, but do not require it. The usual practice is that each presenter informally presents his or her paper to the rest of the participants, and debate it for as long as there are questions which are pertinent to the majority of the group. Presenters should not assume that the other participants are experienced in the particular technologies they will be addressing: for example, not everyone in the audience will be a UNIX system administrator or Smalltalk programmer, or understand terms like jitter, PNL vector and erlang.

Presentations can take anywhere from a half-hour to a half-day depending on the interests of the group, as expressed by the amount of discussion generated. The moderator allows questions during the presentation itself, but they can be clarifying questions only. After the presentation, it is open season and the moderator allows any comments and questions which are pertinent and not offensive. We do not guarantee that all attendees who want to will necessarily present, because a workshop session may run out of time. In keeping with the informal nature of the proceedings, overhead projectors are only used to display related graphics and data. A flip chart or white board is available if presenters want to use it. If a presenter wishes to demonstrate software or show test results on-line, most participants bring their own laptops. TOP

Papers:

 We prefer that each presenter writes a paper on the topic of his or her presentation and distributes it to the other attendees at least one week before the WOPR meeting, though the paper is not mandatory. A paper does not have to be polished or near publication-ready status, but the quality of the papers influences who has priority on the final invitation list. Papers vary from one-page bullet point outlines to 25-page essays. Because everybody is very busy, a lengthy white paper is not required though it is welcome. The minimum we expect is a one-page, bullet-point key-word outline.

We expect all attendees to have read all the submitted position papers and case studies for a session, before the start of that workshop session. This means all papers must be received a week before the workshop session or earlier, so they can be distributed and read before the workshop session starts.

At the end of a session, we encourage participants to form small groups based on common interests, to write and publish joint papers after and based on the workshop session. It is customary in these papers to give attribution to the list of attendees in the session. At least one book was initiated by a workshop collaboration: the bestseller: " Lessons Learned in Software Testing " by Kaner, Bach and Pettichord. TOP

Costs:

 There is a variable per person fee for attending a WOPR workshop (the minimum needed to cover the expenses of a meeting facility and refreshments), and you or your firm has to pay your own travel expenses. At the discretion of the advisory board, the fee will be waived in cases of hardship. TOP

Facilitator and Scribe:

 Each meeting is facilitated by a trained, experienced facilitator who has some familiarity with the subject matter (and usually volunteers his or her services). There is a separate additional role of scribe or note-taker, also a volunteer. This role is described later in the Question & Answer section , Question #7. TOP

Rules of Engagement:

 The LAWST ( Los Altos Workshop on Software Testing ) rules of engagement are the model for WOPR The rules cover in-session conduct, follow-up publications and intellectual property rights.  TOP

Prerequisites:

 There usually are no prerequisites for a session, except to come prepared. Being prepared means you: (a) have your own paper and presentation ready, if you are giving one; (b) have read the papers distributed by others prior to the session; (c) are fresh and well rested; (d) are ready to participate, contribute and learn, with an open attitude; and (e) are ready to make some new friends and have fun. TOP

Organizers and Advisors:

 The organizers include several experienced practitioners with specific interest in performance related topics and and/or significant experience in workshops of this style. The current organizers and advisors are Scott Barber , Ross Collard , Roland Stens, James Bach, Linda Hamm and Doug Hoffman .

The organizers have five main responsibilities:

1. Organize workshop sessions and set the theme or topic for each one.
2. Review papers and select participants.
3. Participate in workshop sessions themselves, when feasible.
4. Inform members by periodic communications.
5. Help represent WOPR in the broader community.

TOP

Representation:

At the discretion of the advisory board, WOPR may elect to represent its members and promote their interests. We may make statements and take positions which could be interpreted as political or lobbying on topics of interest, such as public commentary on proposed standards for performance measurement. TOP

Affiliations:

 WOPR seeks to form mutually beneficial alliances with other organizations. Suitable candidates will share our interests and have integrity. Suggestions are welcome -- please direct us to suitable organization(s). TOP

Timing:

We typicaly hold workshop sessions at least twice per year and schedule each session for three days starting on a Thursday and ending on a Saturday.

The agenda typically is as follows: a welcoming group dinner at 7:00 pm or 7:30 pm on the Wednesday for those able to attend that evening, then 9:00 am to 5:00 pm on both the Thursday and Friday, and 9:00 am to 3:00 pm on the Saturday. In this three-day format, the second day is the high-power day. The first day is the get-acquainted and figure-out-what-the-topic-is-anyway day. The third day is the summarize-and-review, closing ideas, and so-sorry-I-have-to-fly-home-early day, and tends to have lower energy.

People who can attend only part of a workshop session normally will not be invited. If most people cannot be away for three days but can attend for two, we will consider trimming the duration back to two days. We will trim only reluctantly, because going from two to three days increases the workshop session value by far more than 50%. TOP

Sponsorship:

 WOPR by design has no sponsors who contribute financially, just friends who contribute in non-monetary ways. WOPR is scrupulously vendor-neutral. Though if topics like comparative product assessments (“Is Linux faster than Windows on embedded devices?”) are of interest to the participants, there are no inhibitions about discussing them. One wag has suggested that we change the title to “workshop hinging on practical performance endurance & robustness (WHOPPER)”, and try and get Burger King to sponsor it. TOP

Comparable Workshops:

 Some organizations have expressed concern that their busy and valued employees will be attending yet another meeting, and have asked what WOPR offers that is different from existing classes and conferences.

There is no other venue similar to WOPR Few public training classes are available in these areas. The most prominent classes available, in terms of headcounts, are the ones Scott Barber and Ross Collard teach through Software Quality Engineering (SQE). Graduates of classes like this are good candidates for WOPR, generally at the beginner-to-intermediate level. CMG (Computer Measurement Group) conducts conferences on performance measurement but these are not test-centric – the pragmatic issues facing testers are rarely addressed. ISSRE is an annual IEEE conference on software reliability engineering which is too theoretical for many testers. These are a few examples of what is available, and nothing else is comparable to WOPR TOP

Associated Training:

 To help less experienced practitioners prepare for a WOPR session, the organizers will offer a two-day workshop on performance and reliability before each workshop session. Each class will be small, with probably no more than a half-dozen attending. We will endeavor to have subject matter experts team-teach these classes. This training will be held immediately before a WOPR session and in the same location. The classes will be offered at significantly discounted rates from comparable public training. We will use the revenues generated from a class to help offset the costs of hosting the associated workshop session. TOP

Intellectual Property Rights and Confidentiality:

 All participants in a workshop session will make a verbal but legally binding agreement with their peers at the beginning of that session, and covering only the intellectual content of that particular session.

1. No information is communicated which requires the signing of non-disclosure agreements (NDAs).
2. We recognize that participants in a particular session will jointly develop common work products. These are owned by everyone but are not exclusively owned by any one individual.
3. Anyone within this group of participants can re-use any of the work products developed in this session without asking further permission than the verbal agreement.
4. One exception to unlimited re-use is plagiarism. For example, one participant cannot re-publish under his or her own name the position paper that another person had distributed prior to the session.
5. Another exception is the publication or mention of company names without written permission.

Any use of the jointly developed work products requires attribution to the full list of participants in that workshop session. TOP