Thanks to AI detect what really matters in changing situations

Frequently-used practices such as collecting public opinion, understanding consumer satisfaction, and checking on the state of affairs are matters that fall into the realm of market research. And when thinking of ways to conduct research to analyze public sentiment, traditional methods are usually the first to come to mind – like surveys, polls, and focus groups.

But the reality is that these traditional market research methods are slow – and costly. An entire survey project can take weeks to complete. For a phone survey, the average time is 4 to 8 weeks from start to finish; for email, it’s even longer with at least 6+ weeks needed. Online surveys seem to make the most sense these days as people are more technologically forward – but even those can take anywhere from 2 days to 4 weeks to complete depending on the survey size and deliverables.

We’ve heard from organisations frequently, who have growing frustrations about the limits of surveys. Last year, one of our friends at a global NGO explained how their gender education campaigns were informed by surveys. Every 3 years, the NGO runs a survey in 2 cities (of the 8 countries in Latin America in which they will run the campaign). They do fantastic, in-depth surveys in these 2 cities; they must then extrapolate those conclusions to the different cultures and countries where they work. After that, they create their education program based on what they learned, and run that program for the next 3 years. But what happens during those 3 years? ‘During that time, we are blind!’. During that time, numerous local events or trends appear – whether it is a famous legal case about women’s rights, a new political movement that triggers anti-women sentiment, or a pandemic which causes more women to be locked in doors with abusive partners. 

These frustrations are becoming ever more apparent during COVID-19, where needs are changing week to week. In another conversation with a governmental organisation last week, we heard “We normally collect this information through surveys, that take time, and a lot of politics and work. Also in such a dynamic an evolving situation as the Covid response, I need real time info”.

So, what happens when government and public institutions need to make decisions that affect citizens quickly and efficiently? Say, for example, during times of a natural disaster, extreme racial divide, or global health pandemics such as the one we’re currently facing with COVID-19?

This is where a platform that uses Artificial Intelligence and Natural Language Processing is a game-changer.

The capabilities of AI and machine learning are simply far superior to traditional research methods. When you consider coverage and volume –  AI provides  hundreds of thousands, or millions of responses per month from which to extract data versus traditional sample sizes of hundreds or thousands; instead of one city, hundreds of cities can be included.

In terms of staying current and relevant, AI-driven NLP technology goes beyond providing a static snapshot in time – which is what a survey provides. Because it’s all in real-time, tracking how the situation evolves over time – and adapting policies and initiatives accordingly – becomes possible.

AI also has the clear advantage of time and efficiency. Rather than waiting weeks to extract actionable insights from data, trends can be detected in days. Not to mention the type of information you end up getting by listening to what’s happening among a population. With a survey, you only get answers to fixed questions –  the ones that are asked. Whereas, this way, answers to questions you never even thought to ask come to the surface.

That’s why AI is already being used throughout Europe to address concerns during the COVID-19 pandemic. Citibeats has launched collective intelligence ‘observatories’ in more than 27 countries to bring real-time views on social needs to governments and decision-makers. As we’ve all seen firsthand, the dynamics of this global phenomenon is constantly changing and fast, relevant data is needed in order to keep up. By analyzing millions of opinions every week, leaders can use “citizens as a sensor” to respond to top-priority needs that are unmet or looming on the horizon. 

What we’re seeing is that surveys and AI text analytics can act as complementary tools in the toolbox. AI text analytics, as this real-time, with tremendous coverage across geographies and topics, is like an early warning sensor seeking out new issues; surveys, on the other hand, are ideal for validation on particularly important topics that are detected. This is how we see our clients combining the best of both approaches: Citibeats for detecting issues earlier, and then using the limited resources to run surveys to validate the topics which really matter most.

While we see AI text analytics and surveys sitting together comfortably as complementary ‘tools in the toolbox’, it is clear that this is part of a wider paradigm shift. This shift, triggered by abundant public data on people’s opinions, is the direction that research is moving, towards the new field of social data science. We believe that by pushing to innovate in AI text analytics, for example by leading in the area of demographic representativity, or creating indexes that enable benchmarking, AI text analytics is a tool that organisations that seek to understand people in fast-changing environments will be able to rely on throughout their research cycle.