Jump to content

Developer Satisfaction Survey/2024/Shared test environments

From mediawiki.org

πŸ“–Β Developer Satisfaction 2024 Report

πŸ§ͺ Shared test environments

[edit]

tl;dr

  • Use – A majority of respondents used a shared testing environment
  • Beta – 45% are satisfied with beta, up from 38% in 2023
  • Pre-merge – 50% are satisfied with their ability to test in a shared environment pre-merge
  • Priorities – respondents ranked their shared testing environment priorities


# Shared test environment use

We asked survey takers, β€œIn the past year, have you used any shared test environments (e.g., beta cluster or patch demo)?” A majority of respondents (54.5%) said they had not used a shared test environment in the past year.

top


# Shared test environment use (by role)

top


# Shared test environment use (by tenure)

top


# Beta satisfaction

We asked survey takers, β€œIf you have used the Wikimedia beta cluster, how satisfied are you with the beta cluster?”

Of those who said they had used the Wikimedia beta cluster:

  • 45% said they were satisfied
  • 18% said they were neither satisfied nor dissatisfied
  • 32% said they were dissatisfied
  • 5% said they were unsure

top


# Beta satisfaction (by role)

top


# Shared test environment satisfaction

We asked survey takers, β€œHow satisfied are you with your ability to test your code in a shared environment before merge?”

  • 50% said they were satisfied
  • 11% said they were neither satisfied nor dissatisfied
  • 34% said they were dissatisfied
  • 5% said they were unsure

top


# Shared test environment use cases ranked by priority

We asked survey takers, β€œHow would you rank the following use cases for a shared test environment? Rank the following from 1 (most important) to 5 (least important).”

Respondents were asked to order each of the following use cases by priority:

  • Exploratory testing/QA of recent patches
  • Demoing/showcasing
  • Test automation of end-to-end tests
  • Test automation of integration tests
  • Test automation of unit tests
  • Other

These were presented in a random order to each respondent.

The chart below shows descending order of average rank. Like last year, "Exploratory testing" has the highest average rank and received the most #1 rankings. The use cases with the average lowest rankings were "test automation of integration test" and "test automation of unit tests". The graph does not capture "other".

top


# Feedback

We asked survey takers, β€œPlease share any other feedback you may have about shared test environments” Some themes emerged from the answers.

Folks like PatchDemo

PatchDemo is a great tool, and it would be great if it could be improved to work for more code bases and use cases.

Beta

Beta is useful for finding terribly bugs that would knock over production, but that's about it. Most of the "interesting" real world test cases for our code involve user content, and that's not present on beta and painful and difficult to import.