Artificial Intelligence diffusion in the Labour Market, a machine learning approach

The debate around Artificial Intelligence (AI) induced automation and the labour force is extremely vivid, but rarely based on strong empirical evidence.
How fast is AI diffusing in industry and services? A few academic studies propose frameworks to estimate the impact of the fourth industrial revolution on the labour market but they reach no consensus. Indeed, data is scarce when it comes to Artificial Intelligence workers and actual implementation of AI systems within companies. The truth is we have no comprehensive statistics that describes AI diffusion within companies and societies and current estimates are based on qualitative data and perception surveys from astonishingly small samples.
What’s more, the European Union and governments across Europe are currently devising their own AI strategy, craving for more objective data and insights about AI progress and hindrance.
We suggest LinkedIn insights can provide answers to these questions. By scrutinizing AI talents, their skills, the industry they work for, but also the vacancies advertised on our platform we can provide the first building block of a more comprehensive piece of research to better inform the debate about the impact of AI on the Labour market. Identifying AI specialists among the hundreds of petabytes of LinkedIn database reveals a “needle in a haystack” type of problem. To distinguish AI specialists who actually implement AI, from other members with a mention of AI in their profile, I built a text classifier using Scikit-learn and NLTK. This discussion will describe the methodology I developed and the challenges I faced to transform LinkedIn big data into actionable statistics to be used by policy makers.

Thomas is an Economist and Data-Scientist at Microsoft and LinkedIn. He digs into data to study labour mobility, skills’ evolution and the future of work. Before joining Microsoft, Thomas held different positions within the French administration, as a researcher and statistician then as data officer, fostering the use of big data for policy making. He collaborates with the United Nations and served as a visiting research fellow at the Human Development Report Office in New York. As a research fellow with Data-Pop Alliance he gives data literacy and code training to students and UN staff. His ambition: democratizing data-science for social good.