This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .
In 2021, nearly three-quarters of employees experienced a 70% increase in meetings after organizations shifted to remote work due to the onset of the COVID-19 pandemic.¹ When 43% of remote workers state they do not feel included in meetings², the natural question is, how can we make meetings more effective and inclusive?
We were curious too, so we decided to survey a large number of Microsoft customers who wanted to improve meeting experiences in their organizations. As it happens, this was the first large-scale study that we’re aware of conducted by a technology company to determine what makes meetings effective and inclusive.
We used statistical, mathematical, and machine learning techniques to analyze meeting dynamics through survey data and anonymous telemetry. This enabled us to build a model to understand and predict which meetings would be effective, and why. Criteria such as using an agenda, active participation, having video turned on, keeping meeting sizes small, and sharing pre-meeting material all ranked high for driving inclusiveness and effectiveness. Additionally, data analysis showed strong connections between meeting participation and attendees’ subjective perceptions of inclusiveness, sense of comfort, and meeting effectiveness.
Since meetings are a permanent fixture of modern work, we encourage you to dive into the findings below and find new ways to promote behaviors that lead to more effective and inclusive meetings.
Identifying Our Audience and Survey Methods
We used two survey methodologies in tandem to measure meeting effectiveness and inclusiveness across Microsoft employees and a select group of Microsoft Teams customers:
- Post-meeting pop-up - A two-question survey within Teams that popped up at the end of a meeting to collect sentiment in near real-time.
- Email-based survey - Approximately 20 questions designed to understand the nuances of the most recent meeting the respondent attended.
The in-Teams survey methodology was designed with goals to:
- Reach a random sample of meetings throughout the organization
- Minimize intrusion in the workday
- Lower the burden to respond
- Keep responses anonymous to ensure a psychologically safe place to provide feedback on meeting effectiveness and inclusivity
Survey types and population groups have different survey response rates, adding another layer of sampling to our data collection. While this additional layer induces a "non-response" bias to the data, our analysis showed a negligible amount of bias that did not degrade the data utility.
Modeling the Survey Data
To understand the drivers of meeting effectiveness and inclusiveness, we began examining the strength of dependencies between user ratings on these criteria with expected factors including meeting duration, the number of participants, usage of audio/video/screen sharing, and presence of an agenda among others.
We used a variety of statistical tests and machine learning (ML) algorithms to learn the set of factors and combinations that correspond to fundamental differences in ratings of effectiveness and inclusiveness. While an abundance of articles and small-sample studies provide a long list of potential drivers for meeting effectiveness and inclusivity, many of them are shown to not be significant after large-sample analysis.
A proven method of breaking down the collective pattern of dependencies among the set of potential factors and ratings is through the use of graphical models. We developed a process for selecting top contributors by fitting a web of dependencies and estimating the strength of each connection. A sample of such a model is shown below. The following graphical model from our peer-reviewed publication³ illustrates the multivariate interplay between factors.
The arrows in the above graph show the relationships between the variables and in relative terms. Using green arrows to show positive correlations and red arrows show negative correlations, this model helped us select the truly impactful factors on meeting effectiveness and inclusiveness from a sea of claims and anecdotal hypotheses. For example, the green arrow with “2.5” between “Participation” and “Inclusive” can be interpreted as meetings with high participation are 2.5x more likely to be rated as inclusive.
Participation represents "speaking in the meeting more than once." It shows a significant positive impact on the chance of the meeting being rated inclusive (4 or 5-star). This pattern consistently shows up regardless of the survey type or audience group. The strength and consistency of this impact encouraged us to develop targeted ML models to further understand details about this correlation. Specifically, to include the interaction effects and explore the ramifications under different segments. Our findings show that:
- Meetings with higher levels of participation were rated more inclusive. Participants who spoke often during the meeting gave a 98% inclusive rating, participants who spoke a few times gave an 89% inclusive rating, participants who spoke only once gave a 67% inclusive rating, and participants who only listened gave a 36% inclusive rating.
- Participants who felt their presence was necessary for the meeting felt more included – giving the meeting a 92% inclusive rating over those whose presence was not necessary, who gave a 53% inclusive rating.
- Meetings with smaller meeting sizes were rated as more inclusive. This is a decreasing trend where 2-person meetings were rated by all participants as inclusive, while meetings with more than 10 people were rated only 60% inclusive.
Our analysis also showed that the impact of participation on inclusiveness was also influenced by other factors including meeting duration and length of the meeting. For example, the most significant gain is observed for thirty-minute or shorter meetings. In this segment, participation can increase meeting inclusiveness by ~6% and effectiveness by ~4%.
Video usage was another area where the large sample of our survey allowed us to learn from nonlinear patterns. While additional research is warranted to understand the breadth of meeting scenarios that benefit most from having video on, and which scenarios benefit from having video off, our survey data did yield interesting findings. We found the meeting scenario with a higher chance of benefiting from video usage is a "small meeting” with fewer than eight participants where some people are in a meeting room, and the rest are joining remotely. The benefit mostly comes from local participants (those in the meeting room) being able to see remote participants via video. Larger meetings may be more prone to “Video Conference” due to prolonged eye gaze, an effect that’s magnified when users are required to stare at numerous one-inch boxes of faces on their screens.⁴
More Resources and Insights
In addition to encouraging behaviors correlated to increased meeting effectiveness and inclusiveness, Microsoft Productivity Score and Viva Insights are valuable resources to help monitor organizational productivity, effectiveness, and engagement.
Better Meetings… Through Science
Bookshelves are full of pages dedicated to the topic of meeting effectiveness, though many of these insights are based on small samples and anecdotes. While there’s still much about meetings yet to explore, such as the influence of meeting purpose on effectiveness and inclusion (brainstorming and problem solving versus status meetings), our research offers valuable constructs for how to make such connections.
What is clear in a hybrid-first world, is that the volume of scheduled and ad-hoc meetings is likely to increase. Organizations need a trusted methodology to identify and measure the factors that lead to more valuable meetings and enable us to form and strengthen connections between colleagues and friends.
1. Atlassian. 2022. You Waste A Lot of Time at Work Infographic | Atlassian. [online] Available at: https://www.atlassian.com/time-wasting-at-work-infographic
2. Work Trend Index: Microsoft’s latest research on the ways we work
3. Cutler, R., Hosseinkashi, Y., Pool, J., Filipi, S., Aichner, R., Tu, Y., & Gehrke, J. (2021, February 19). Meeting effectiveness and inclusiveness in remote collaboration. arXiv.org. Retrieved May 23, 2022, from https://arxiv.org/abs/2102.09803
4. Karl, K., Peluchette, J. and Aghakhani, N., 2021. Virtual Work Meetings During the COVID-19 Pandemic: The Good, Bad, and Ugly. Small Group Research, 53(3), pp.343-365