By: Rachel Kline
Originally published in Authority Magazine on August 13, 2023.
In today’s competitive market, delivering an outstanding user experience (UX) is critical for product success. A well-designed UX can lead to higher user satisfaction, increased engagement, and ultimately, improved brand loyalty. But how can product designers, developers, and organizations create user experiences that truly stand out and make a lasting impression on users? In this interview series, we are talking to UX professionals, product designers, developers, and thought leaders to explore “The 5 Best Ways to Elevate Your Product’s User Experience.” As part of this series, we had the distinct pleasure of interviewing Ann Aly.
Ann Aly is a researcher with a background in academic social sciences. After leaving academia, she pivoted to service-based research in non-profit and civic tech. She is currently the Director of HCD at TechFlow, where she works with government agencies to modernize their approach to user experience.
Listen to your users. I mean this in the purest definition- listen without judgment and temper your assumptions about what they might say. Let them talk about their experiences and be open to twists and turns you may not have expected. Indi Young’s Time to Listen has an excellent framework for this.
Thank you so much for joining us in this interview series! Before diving in, our readers would love to learn more about you. Can you tell us a story about what brought you to this specific career path?
I was in a rough spot professionally, because I realized that working in higher education and non-profit were not good fits for me. I wanted to continue to conduct service-oriented research, so I began looking into ways to do that in private industry. As I was job hunting, I was also providing end of life care for my mom, who passed away from cancer in 2019. My mom was a dual Medicaid and Medicare beneficiary, which made medical care possible for her as a disabled, low-income person. However, it was also the source of a lot of paperwork and confusion.
About a month after my mom passed, I saw an opportunity to work on a contract with the Centers for Medicare and Medicaid Services to conduct research on various burden-heavy services- one of which was the dual Medicare/Medicaid program. I immediately applied and pivoted both my research skills and my lived experience of government services into my first civic tech role. I’ve been in the field ever since and am so happy I made this switch.
Do you have any mentors or experiences that have particularly influenced your approach to product development and user experience?
Yes! Here are two examples:
1. Lived Experience: I grew up low-income and depended on many government services for basic needs. The people who rely on these services are the experts of their needs and experiences- not the people who create or design the services. Navigating eligibility paperwork for things like SNAP, immigration visas, health insurance, and child support can be such a burden that eligible folks will often go without these services due to paperwork mistakes or confusing instructions. I keep this in mind when conducting research so that I’m centering them as the experience expert and myself as the nerd who will translate what they need into a service that makes it easier for them to get what they need.
2. My current TechFlow teammates: I’m in an odd, but hugely beneficial situation at work where the team that I lead is made up of senior civic tech researchers who all have more experience than I do. I learn so much from them and it has helped shape my leadership style to that of a facilitator (basically- how can I advocate for you and get blockers out of your way?) instead of a hierarchical chain of command. It’s amazing how a more communal approach ensures that everyone is heard, valued, and that we can swarm together to accomplish more than any one of us could do individually.
It has been said that our mistakes can sometimes be our greatest teachers. Can you share a story about the funniest mistake you made when you were first starting? Can you tell us what lesson you learned from that?
It’s more of a habit than an isolated mistake for me, but I tend to hyperfocus on tasks. This can be beneficial to deep work, but it also can lead me down some rabbit holes. Before I knew how to manage this tendency, I had the habit of doing a deep dive into a topic or task and emerging with 20 pages of notes, a conspiracy board, or an Excel spreadsheet with ten slicers and my teammates would be like “Ann…we asked for an outline”. Now that I know about my tendency to do this, I always timebox my first draft so that I can get feedback before I go too far into something. I have learned to work with the tendency, not against it, and that’s helped. I know my first draft of anything will likely be unhinged, so I plan for it now.
What do you feel has been your ‘career-defining’ moment? We’d love to hear the lead-up, what happened, and the impact it had on your life.
This may seem atypical, but here we go: When I was doing my dissertation research in the Basque Country, I was recording interviews between people who spoke Basque and Spanish. One of the interviews was between a woman in her 20s and her grandma. Due to some sound quality issues, I ended up not being able to use the recording in my analysis. After the session, the young woman followed up with me and asked if she could have a copy of the recording because her grandma spoke a rare dialect of Basque that is only heard in rural speakers of her grandma’s generation. I sent it to her (of course) and really stopped to think about how what was an “unusable” recording for me was a piece of history for this woman’s family. It reminded me to zoom out when doing research (there’s that hyperfocus/myopia again!) and be more mindful about how the research I’m conducting impacts the participant’s lives and how that work may help them beyond whatever my objective or hypothesis is.
This is not a new problem or novel problem in social science, where academic protocols tend to be extractive and consider their participants second. However, that was the moment that it happened “in the wild” for me and challenged me to consider the impacts of my research first and not as an afterthought to my objective. I didn’t want to continue the tradition of extracting data from a community and then just moving on when my paper was written.
I’m still in touch with the woman from the Basque Country- we became good friends after and I dedicated my dissertation to her.
Can you tell us a story about the hard times that you faced when you first started your journey? Did you ever consider giving up? Where did you get the drive to continue even though things were so hard?
One of my first projects in civic tech was complicated by two factors: the political appointees as the agency we were working with were not happy that some of our results went against their campaign/party beliefs and the start of the COVID pandemic. This not only changed the scope of the work that we were doing (in-person field research was now off the table) but also how our work was being received by that agency’s leadership. It was really discouraging because we were uncovering insights that could have been majorly impactful for policy changes, but many of them were deprioritized. I had to remind myself that not every project would be like this and that it was important to be calmly persistent with evidence-based findings when faced with resistance and faulty assumptions. Sometimes unexpected results can be shocking for people and adopting new approaches to old problems can be hard, so giving people time to “mourn” their old understandings to make room for changes is an important part of reporting new findings.
Can you share any strategies you have for effectively gathering and analyzing user feedback?
For gathering, I like to encourage users to describe examples of the problems they are trying to solve that go beyond the product or service in question. I like to learn the context that the product/service exists in and how it may (not) interact with other pieces of technology or equipment. I’m also fascinated by workarounds, so I like to observe a person’s process to see the unusual or unintended ways they interact with something.
For qualitative analysis, I like to let patterns emerge. I may use the topics of the questions as an initial way to group the data points, then I start looking for connections, commonalities, outliers, or whatever other descriptor helps me understand the data and what it can tell us.
For quantitative analysis, I have no special strategies. Just leave me in a room with a spreadsheet all day and I will emerge with something*. This is where my hyperfocus shines.
*That something may need a round 2 of analysis after my team holds me accountable, but it’s usually a good start
Can you share an example of a time when you received user feedback that prompted you to make significant changes to a product’s user experience, and how did you approach incorporating that feedback into the development process?
I was working on a project that involved employees making service requests for things like office repairs and tech equipment. When we spoke to the employees who used the system, they reported that it was easy to use and didn’t have any issues with it. Upon observation, we saw that most participants used the “other” option on the service form instead of the existing categories (repair, monitor, mouse, lamp, etc). This workaround wasn’t seen as a bother by users, but we wanted to check with the service providers and get their input on this. We learned from the service providers that whenever a requestor chooses “Other”, this creates delays in routing that request to the correct person/department and impacts their response time metrics. This made it clear to us that the request form needed to be redesigned with better instructions and options before it was released so that requestors were more likely to avoid the “other” option (except for truly unique scenarios) and make the fulfillment time quicker. This was a good lesson for us in the importance of researching the service recipient AND service provider to have a better understanding of the experience.
How do you balance the need for simplicity and ease-of-use with more complex or advanced features in a product, and what strategies do you use to make sure that users can navigate those features without getting overwhelmed?
This balance is interesting for government services because you’re legally required to comply with a standard of accessibility and ease of use (called Section 508). This means that any complexity incorporated into a service still has to be accessible to all users. We do two kinds of testing to see if we’re meeting this goal- specific accessibility testing, completed by someone certified in 508 standards and usability testing with real users to see how they interact with different facets of the service in question.
Another interesting part about government services is that complexity and innovation typically aren’t the end goals- it’s more about access for recipients and burden reduction for providers. This impacts the way more advanced features are considered, because they still need to fulfill a need that the public has, as opposed to a “nice to have” that you may see with consumer services. Since 2021’s Executive Order on customer experience was released, government agencies have put more resources and priority into improved user/customer experience, including plain language instructions (in multiple languages), help in multiple modalities (phone, chat, in-person), and mirroring user interfaces that are common in private industry. There’s a long way to go, but the progress that’s been made so far to make common services (like taxes, applying for a visa, qualifying for healthcare) less of a labyrinth for everyday people. One of the examples that demonstrates this is the work that TechFlow did with partnering firm Coforma on together.gov, which provides pathways to reunify families separated at the southern US border between 2017–2021. The website’s content is available in 7 languages (including audio), to account for a range of literacy levels and hearing capabilities. The development team also partnered with immigration NGOs to work directly with real people impacted by US immigration policy to validate that the site was relevant and usable. This site scored 98/100 for accessibility (via Lighthouse) and has facilitated 3,033 successful family reunifications since September 2021.
What are some of the strategies you’ve used to make your product ‘stickier’ and increase user retention?
For simple retention (ie use the product/service and stay with it), I think it’s important to make sure that users are heard and that they know they are being heard. Tell users why you want their input and how their input contributes to the direction of the product or service. Legitimately make them a crucial part of the journey where no major decision is made without looking at how it will impact users and what they need.
I also like to think about overall workflow improvements, since many government services are (one of) the only providers of their offerings or make global changes to how services are delivered. In these situations, retention isn’t really a choice, but workflows and efficiencies may be impacted by the quality of the service. For example, the US Army is changing all its non-tactical vehicles to electric vehicles over the next several years. The Army doesn’t need to worry about retaining electric vehicle drivers, since this is not optional for service members. However, the TechFlow team has published a research-based blog encouraging them to consider how the adoption of electric vehicles will impact someone’s workday, timeliness, and quality of service provided with the new vehicle. Even if adoption of a product/service is optional, there are still crucial portions of the user’s experience that need to be examined to make sure that the adoption process does not have negative ripple effects on the systems it impacts.
In your experience, what are some of the most effective ways to measure the success of a product’s user experience, and how can this data be used to continuously improve the product over time?
Let the data tell this story and there’s a couple different ways to do that:
1. Look at the usage patterns: Collect baseline and interval data points on how a product or service is being used and how those change over time (or after something is released/discontinued. Some insightful usage variables can include number of total users, number of returning users, average length of session time, and level of engagement (ie does the user interact with multiple facets of the product or service?)
2. Ask users what they think: Sentiment and usage can often be at odds! There are cases where someone can be a highly engaged user but also have negative sentiment towards the product or service. By asking users how they experience the product (is it useful? Is it easy to use? What are their challenges? What problem(s) are they solving with it), you get a more complete picture of the user’s experience beyond a purely quantitative measure of usage
3. Observe usage: This is a neat mix of usage data and user sentiment and works really well for learning about workflows. By observing how a user integrates a product into their daily life, you’re able to gain context on things that sentiment and usage alone cannot reveal. For example, many users have workarounds (ie not using a product in its intended way) that are hard to describe or may not seem like unintentional uses. By observing a workaround and asking questions about it, opportunities for improvement may arise that would be harder (or impossible) to glean from other success measurements.
Thank you for all of that. Here is the main question of our interview. Based on your experience, what are your “5 Best Ways to Elevate Your Product’s User Experience”? If you can, please share a story or an example for each.
1 . Listen to your users. I mean this in the purest definition- listen without judgment and temper your assumptions about what they might say. Let them talk about their experiences and be open to twists and turns you may not have expected. Indi Young’s Time to Listen has an excellent framework for this.
2 . Observe your users. There are so many things that are difficult or impossible to explain verbally. Observing a user’s experience “in the wild” will give you insights that listening or asking questions alone cannot. For example, my team would have never known about a workaround that folks were using on a service request form (choosing “other” as a default and typing in the issue) if we hadn’t directly observed them submitting a request!
3 . Be willing to shift gears in the face of new findings. It’s okay if your hypothesis isn’t supported, your results are surprising or inconclusive, or there’s a huge change in the world (like a pandemic!). It can be stressful to make changes after time and resources have been dedicated to the previous plan, but the point of UX isn’t commitment to a plan. It’s commitment to your users and serving them.
4 . Learn how to talk about impact. Just because you are enthusiastic about user experience doesn’t mean that everyone else will be too. You may need to educate others to get them invested in UX and why it matters. Even if not intentionally, you may encounter some resistance or skepticism and learn how users of your data can get the most value out of the work you’re doing. Non UX folks want to see what (if any) difference your work has made- this can be anything from an increase in a certain user type, faster application times, fewer help desk calls, positive user testimonials, or increased user trust.
5 . Accessibility. Don’t make this an afterthought! Take the time to learn how (not if) your product or service can make improvements to be more accessible to different ability levels. Take into account things like vision, hearing, cognitive abilities, language comprehension, age, physical limitations, and cultural variables to audit how you can better serve these folks. You won’t know until you actually take the time to research this and incorporate it into your process.
We are very blessed that very prominent leaders read this column. Is there a person in the world or in the US with whom you would love to have a private breakfast or lunch, and why? He or she might just see this if we tag them 🙂
Sam Ladner- your book on mixed methods is a crucial reference to me when transitioning from academic to industry research and I appreciate how clear your writing is!
Thank you so much for this. This was very inspirational, and we wish you only continued success!