CHI 2021: Making remote and hybrid meetings work in the new future of work

This post has been republished via RSS; it originally appeared at: Microsoft Research.

Over the course of the COVID-19 pandemic, some truths about the nature of work have been underscored: it is uniquely complex, quickly shifting, and increasingly technology-mediated. Teaching, medicine, mental health, and other professions—previously thought to be near-impossible to do remotely—have all abruptly moved to online and hybrid mediums. All kinds of workers have needed to find new and creative ways to do their jobs. For many, the boundary between office and home have become a thin, blurred line. These changes have not been small, and they have not been without growing pains.

Situated in this landscape, researchers are now tasked with the challenge of trying to understand how these short-term transformations will impact workplaces and work cultures in the long term. Taking place a little over a year after many quick-pivot workplace changes unfolded, the 2021 ACM Virtual Conference on Human Factors in Computing Systems (CHI 2021) is a timely moment for researchers to present some of their initial findings about how workplaces and people have adapted this past year, how technology has helped to aid in these transitions, and where technology needs to grow to support a new future of work.

A look at a cross-section of research from Microsoft accepted and presented at CHI 2021 reveals that among these changes to the workplace, researchers are also thinking about the important role that frameworks for interpretability and privacy will play in the workplaces of the future—especially as they pertain to AI becoming a more effective tool for workplace insights and collaboration. Some CHI papers confront interpretability and trust head on, like “AI Trust Score: A User-Centered Approach to Building, Designing, and Measuring the Success of Intelligent Workplace Features” and “Manipulating and Measuring Model Interpretability,” while some have a narrower focus on how AI might impact certain fields, as in “Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens.

One concrete subject seems to be at the crossroads of the many elements mentioned above: remote meetings. Recent findings in the Work Trend Index show that remote meetings are both a growing pain point for companies and workgroups as well as an effective outlet for collaboration between close teams during a period of isolation. The difference between a good remote meeting and a bad one could be the difference between a breakthrough idea or a breakdown in communication, and the technologies that enable these meetings are relying on recent advances to improve meeting quality for teams and individuals. When cutting-edge technology is employed as an environment for and facilitator of these ubiquitous workplace encounters, it’s important that it becomes a tool for streamlining, rather than impeding, a core piece of daily work.

Below, explore three research papers that examine remote meetings and presentations from both human and technology perspectives. One common theme from research into work this past year is that tools need to support not only the work itself, but also the well-being and growth of the people doing the work, and that sentiment can be found throughout these papers. As the move to a more flexible model of work becomes more widespread, findings show that there is plenty of room for imaginative solutions to improve the remote and hybrid meetings of the future.

To learn more about the broader range of work presented by Microsoft, a Champion sponsor of this year’s CHI event, check out the Microsoft Research at CHI 2021 page.

Playing to an audience: Bridging the gap between online and in-person presentations

System architecture and main processing steps of Affective Spotlight. An artistic rendering of a computer monitor shows faces of people in a remote meeting program. Above the faces, a label reads “audience.” There are seven emoji-type faces, in two rows, showing different reactions: heart eyes, concern or skeptism, expressionless, excitement, smiling with close mouth, smiling with open mouth, and surprise. Next to the last face is a drawing of a robot head, representing the technology sending to cloud. 1. Text reads: send audience video streams to cloud services, which points to a box with the excitement face and features being outlined by a yellow box. This points to 2. Extract features from frames (face and landmarks, facial expressions, head gestures). 3. Compute spotlight scores. This points to an image of the excitement face with a spotlight graphic in the upper corner of the box. 4. Share spotlighted audience video stream. This points to an artistic rendering of a presenter speaking to a screen. On the left half of the screen, the presentation shows a pie chart. On the right hand of the screen, the spotlighted audience reaction is shown.
Figure 1: Overview of the system architecture and main processing steps: 1) the video of audience members are captured with a Microsoft Teams bot, 2) video feeds are analyzed to extract affective information in real-time, 3) scores for each audience members are computed and accumulated over a 15-second window, 4) the audience with the highest score is shown as the video feed of the bot.

Ten-second take from the researchers: Current online presentations provide limited to no audience feedback, making it very difficult to establish rapport and effectively adapt the content of the presentation. Inspired by cinematographic efforts, we created AffectiveSpotlight, a research bot built for Microsoft Teams that analyzes facial responses and head gestures of audience members, and dynamically spotlights the most expressive ones.

Diving deeper: The work presented in “AffectiveSpotlight: Facilitating the Communication of Affective Responses from Audience Members during Online Presentations” leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals during online presentations. Using an exploratory survey of 175 respondents, the researchers assessed the most relevant audience responses. These included confusion, engagement, and head nods among others. The team then implemented AffectiveSpotlight, a research bot built for Microsoft Teams that analyzes facial responses and head gestures of audience members and dynamically spotlights the most expressive ones.

Novel angle: Researchers have explored a wide variety of methods to facilitate audience feedback to online presenters. However, the majority of the methods usually rely on audience self-reports which are then provided to presenters in the form of flying emojis or abstract visualizations. In contrast, this work analyzes the facial responses and head gestures of audience members in real-time and explores providing the information in the form of original video feeds, which are more familiar and pose fewer cognitive demands to the presenter as opposed to aggregated data visualizations. The researchers purposely decided to avoid labeling the inferred responses to empower presenters to make their own personal interpretations based on the context and their experience.

Key findings: When evaluating the experience of different presentation systems in a within-subjects study with 14 groups and 117 participants, the researchers observed several statistically significant differences. In particular, presenters using the AffectiveSpotlight rated to be more significantly aware of their audience, spoke for a longer period of time, and self-assessed the quality of their talks more similarly to the audience members, compared to two control conditions (randomly-selected spotlight and default platform UI).

In a user study conducted by the researchers, 83.11% of the participants reported missing relevant audience feedback when presenting online.

Insights from users: The researchers conducted a user survey, where people reported that they would like to see confusion and engagement states more than other cognitive states. Also, participants reported wanting to see head nods more than other audience behaviors. In addition, presenters were less interested in seeing negative states, such as sleepiness or sadness, as compared to more positive states, such as engagement or excitement. This is possibly because negative feedback may indicate lack of audience interest, which might induce performance anxiety associated with giving formal presentations. After using AffectiveSpotlight, both presenters and audience members reported improvements. One presenter said, “AffectiveSpotlight helped create a feedback loop with the audience,” while an audience member compared the effectiveness of AffectiveSpotlight to another method, “When I was not convinced about the cons [of a particular slide] in the first talk [using AffectiveSpotlight], I felt the presenter spent a lot of time trying to get their point unlike the second talk [using another method].” Check out the paper for more feedback from users and how the team conducted its surveys.

Potential future investigations for the research community: While the results of the study were very encouraging, it is important to note that the experiment was performed in controlled settings with a limited sample size. More research needs to be done in less controlled settings to better assess the generalization of the findings to more varied real-life online meetings. In addition, this work focused on the analysis of non-verbal cues to inform the spotlight selection, but future efforts may consider a multimodal approach that combines multiple sensing modalities (for example, microphones and eye tracking) to help improve the quality of the information provided.

The research team includes: Javier Hernandez, Daniel McDuff, Kael Rowan, Jina Suh, and Mary Czerwinski of Microsoft Research, along with Prasanth Murali, PhD Candidate at Northeastern University, who was a Research Intern with Microsoft Research at the time of the work.

MeetingCoach: Personalized post-meeting dashboard for contextual and behavioral insights

Meeting Coach dashboard representation. An application window reads “Meeting Insights: Attendee-1.” The left half of the window is broken down into three vertical thirds. In the top third, A pie chart shows participation as a breakdown of percentages. A comment box reads: “Engage with others: While speaking is not always needed, sharing your agreement on the agenda can make the meeting effective.” In the middle third, A pie chart representing your sentiment shows “positive” and “negative” percentages. A comment box reads: “Keep up the positivity: Continue cultivating a welcoming environment for the team.” The bottom third shows your participation history by meeting. A line graph is displayed in this third with four points that are gradually descending. On the right half of the screen, from top to bottom: A video feed shows video and timestamp of the meeting, A transcript gives line-by-line transcription with timestamps, a continuous timeline showing “team” versus “you” sentiment in blue and magenta spikes and valleys, a timeline indicating when consensus was reached between attendees, a timeline indicating questions, a timeline indicating speaking turn for each individual attendee.
Figure 2: The DashboardUI shows the tested UI for AffectiveSpotlight.

Ten-second take from the researchers: Meetings, and nowadays remote meetings, are an essential part of work, but many employees find them frustrating and even counter-productive when good meeting practices are lacking or violated. We developed an AI system to automatically create personalized post-meeting dashboards for each participant that captured contextually grounded behavioral dynamics. We found that reviewing the dashboard helped improve attendees’ awareness of meeting dynamics, with implications for improved effectiveness and inclusivity.

Diving deeper: The research in “MeetingCoach: An Intelligent Dashboard for Supporting Effective and Inclusive Meetings” involves human-centered design of a post-meeting dashboard to support improving the inclusivity and effectiveness of remote meetings. MeetingCoach is an intelligent post-meeting feedback dashboard that summarizes contextual and behavioral meeting information. Researchers performed an exploratory survey study of 120 information workers to identify important signals (turn taking, sentiment for example) and used these insights to wireframe prototypes. These designs were evaluated with 16 in situ participants who helped identify the components they would prefer in a post-meeting dashboard. After recording video-conferencing meetings of eight teams over four weeks, the team developed an AI system to quantify the desired meeting features and created personalized dashboards for each participant. Through interviews and surveys of 23 individuals, the researchers found that reviewing the dashboard helped improve attendees’ awareness of meeting dynamics, with implications for improved effectiveness and inclusivity.

Novel angle: This paper presents the first implementation of a multimodal (audio-visual) AI system for summarizing contextually grounded behavioral dynamics within remote meetings. In a human-centered manner, the researchers designed a dashboard that supports private reflection on meeting dynamics and shows potential for supporting more inclusive and effective meetings.

Key findings: Simply reviewing the AI-generated dashboard helped improve attendees’ awareness of meeting dynamics. It was easy to review where in the meeting there was consensus, dissent, or confusion. Also, people want meeting feedback at varying times. Sometimes presenting recommended actions and behavioral information immediately before the following meeting in a series would be most actionable for them—rather than directly after the previous meeting.

In a survey conducted by the researchers, 78% of the participants reported understanding social signals to be very important in meetings and 55% of the participants expressed that social signals are more difficult to understand in online meetings.

Insights from users: Many employees find meetings frustrating and even counter-productive, especially when best meeting practices are lacking or violated. The violations of general meeting norms and disrespectful behaviors have been shown to be negatively correlated with meeting effectiveness and satisfaction. The fractured nature of video conferencing can aggravate negative outcomes and marginalize some members of the meeting. Participants also provided recommendations for improving the dashboard. They suggested having fewer and/or customizable features and the need for actionable suggestions that could help them effectively modify their dynamics in the future. Many of these findings could be equally helpful for in-person or hybrid meetings as the goal is to support reflection on meeting dynamics after or in-between meetings.

Potential future investigations for the research community: A primary goal of remote collaboration tools should be to support the most effective meetings possible for all participants. The researchers argue that intelligent, reflective post-meeting dashboards can be useful to help promote positive meeting practices and help raise the awareness of participants about potentially harmful or counter-productive meeting dynamics. The researchers hope that those who seek to build similar meeting support tools adopt their design suggestions:

  • Provide actionable suggestions both after and directly before a next meeting in a sequence
  • Modify feedback design based on its delivery timing
  • Provide training opportunities based on feedback from a series of meetings
  • Incorporate multi-modal signals to provide rich feedback.

The research team includes: Daniel McDuff, Robert Sim, Jina Suh, Kael Rowan, Javier Hernandez, Sean Rintel, Kevin Moynihan, and Mary Czerwinski of Microsoft Research, along with Samiha Samrose, PhD Student at University of Rochester, who was a Research Intern with Microsoft Research at the time of the work.

Analyzing the prevalence and impact of multitasking behavior during remote meetings

Figure 3: Proportion of user-meeting pairs with email multitasking versus meeting characteristics measured by telemetry data. People multitask more in longer meetings, larger meetings, recurring/scheduled meetings, morning meetings, and Tuesdays.

Ten-second take from the researchers: Finding it hard to stay focused during remote meetings and trying to get other things done? You are not the only one! We conducted a large-scale analysis of remote meeting multitasking behavior with US Microsoft employees. A key finding was that the larger the meeting, the more likely participants will multitask, and that isn’t necessarily a bad thing.

Diving deeper: Remote meetings are critical for remote work because of the need for synchronous collaboration in the absence of in-person interactions. In the paper “Large Scale Analysis of Multitasking Behavior During Remote Meetings,” researchers found that in-meeting multitasking is closely related to people’s productivity and wellbeing. In this paper, researchers present what they believe is the most comprehensive study of remote meeting multitasking behavior, through an analysis of a large-scale telemetry dataset, collected from February to May 2020 on US Microsoft employees, and a 715-person diary study from mid-April to mid-August 2020. Findings suggest important best-practice guidelines for remote meetings, some of which include avoiding important meetings in the morning and setting expectations for attention. They also indicate design implications for productivity tools, like support for a scale of attendance attention level and positive remote multitasking.

Novel angle: Both good and bad types of multitasking exist, though we often focus on the negative aspects. The findings suggest it is important to encourage positive aspects of multitasking while reducing its negative implications. Not every meeting requires every participant’s full attention at all times, and productivity tools could help facilitate positive multitasking by:

  • Supporting “focus mode” for remote meetings to eliminate distractions from software interfaces during meetings.
  • Supporting other types of engagement during remote meetings (like note taking through better interface design, such as supporting split views for multitasking).
  • Supporting scaled meeting engagement (from high to low) in invitations and interfaces, to help set attention expectations.
  • Providing features for organizers to rank the importance the full meeting/certain aspects of the meeting to help attendees decide which meetings to attend and when to skip some parts of the meetings.

Key findings: The results demonstrate that intrinsic meeting characteristics such as size, length, time, and type significantly correlate with the extent to which people multitask, and multitasking can lead to both positive and negative outcomes. A few key takeaways include that multitasking happens more in morning meetings and that multitasking can sometimes help boost productivity.

According to the results of this research, around 30% of remote meetings include email multitasking, and around 25% include file multitasking, like working on Word documents.

Human insights: This research shows that both intrinsic meeting characteristics and external factors are correlated with how people multitask during remote meetings. External factors included needing to catch up on other work or anxiety relief among others. People multitask more in large, long, morning, and recurring meetings, as well as meetings with lower relevance/engagement. Also, people multitask for both work-related tasks and non-work-related tasks. Organizers of meetings might consider a few best practices this research puts forward: avoiding important meetings in the morning, reducing the number of unnecessary meetings, shortening meeting duration while adding healthy breaks, setting attention expectations that encourage active contribution and participation when valuable, and allowing space for positive multitasking. The researchers note that it’s important to consider specific worker and workplace contexts when applying these practices.

Potential future investigations for the research community: The researchers identified several areas for future exploration. Since this paper focused specifically on information workers in the US, it would be interesting to learn more about how different worker types and country’s work cultures impact these results. Another avenue may be to compare multitasking behavior in hybrid versus fully remote settings. Building collaboration tools with the design implications laid out in the paper and deeper analysis of whether certain multitasking behaviors were positive or negative also hold potential for further insights.

The research team includes: Shamsi Iqbal, Mary Czerwinski, Sean Rintel, Brent Hecht, Jaime Teevan, and Longqi Yang of Microsoft Research; Hancheng Cao, PhD Candidate at Stanford University, who was a Research Intern with Microsoft Research at the time of the work; Priscilla Wong, PhD Student at University College London, who was a Research Assistant with Microsoft Research at the time of the work; and Chia Jung-Lee, who was with Microsoft Research at the time of the work.

Homing in on how chat is used to multitask during remote meetings

Three people meet in Microsoft Teams. On the left, a person with a beard is wearing headphones with a microphone and is smiling. They wear a black t-shirt, and they are standing in front of a curtained window in a house. A wall to his left has a gallery of pictures. Moving clockwise, another person has long, dark hair and is also smiling. They wear a white t-shirt. They are standing in a room with white walls, and a turned-off lightbulb hangs above their head. Next, a person is talking. They have glasses, a purple shirt on with horizontal stripes, and stand in a room with a bookshelf and floor lamp to their right. There is also another person in a small window at the bottom of the screen, wearing a hat, cardigan, and headphones. The sidebar chat is outlined with a yellow border. From top to bottom, the chat window reads: Meeting Chat. Nancy Baym: “Happy New Years everyone.” A picture of a cat with glasses working at a laptop. Nancy Baym: A video of a dog levitating next to a tree. This video has a “laugh” reaction. A string of unintelligible text in the chat follows. “We need some actual text in here.” Nancy Baym: “Is someone taking notes? Hey let’s have a sharepoint!” A heart reaction shows on the last comment.

The Promise and Peril of Parallel Chat in Video Meetings for Work” finds that simultaneous text, images, files, and links posted in meeting chat areas are now an integral part of remote work meetings. While chat paralleling audio-video has been investigated in other contexts, this is the first study documenting chat in remote meetings for work. It turns out that many remote meetings only work as well as they do because parallel chat is present, and without it some meetings would fail to work at all. As with other multitasking during meetings, parallel chat has pros and cons. Parallel chat can enable more attendees to have a say while avoiding interrupting speakers, as well as a place for detail, affirmations, and even humor. However, it can also be distracting from the main purpose of the meeting and difficult to follow. The researchers suggest potential best practices for managing chat, and they provide design suggestions for improving the display and productive value of chat in products like Microsoft Teams.

The research team includes: Advait Sarkar, Sean Rintel, Rachel Bergmann, Sharon Gillett, Danielle Bragg, Nancy Baym, and Abigail Sellen with Microsoft Research; and Damian Borowiec, PhD Student at Lancaster University, who was a contractor with Microsoft Research at the time of the work.

Discover more from CHI 2021 and the New Future of Work

This is a sampling of work by Microsoft Researchers from CHI 2021 exploring the New Future of Work,  and remote meetings are just one of many aspects of this future that researchers are investigating. If you are still working from home and are curious about how the transition back to the office might go, another CHI 2021 paper out of Microsoft Research Asia, titled “Returning to the Office During the COVID-19 Pandemic Recovery: Early Indicators from China,” explores how knowledge workers navigated the early phases of this transition and the key factors impacting these workers. To learn more about Microsoft researchers’ broader work toward making remote work more effective, check out this collection of resources. For an in-depth analysis of how people’s work lives have changed during the COVID-19 pandemic, you can read the report “New Future of Work: Research from Microsoft into the Pandemic’s Impact on Work Practices.”

The post CHI 2021: Making remote and hybrid meetings work in the new future of work appeared first on Microsoft Research.

REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.