Leveling Up Natural Language Processing with Transfer Learning

Summary

Natural language processing is a powerful tool for extracting insights from large volumes of text. With the growth of the internet and social platforms, and the increasing number of people and communities conducting their professional and personal activities online, the opportunities for NLP to create amazing insights and experiences are endless. In order to work with such a large and growing corpus it has become necessary to move beyond purely statistical methods and embrace the capabilities of deep learning, and transfer learning in particular. In this episode Paul Azunre shares his journey into the application and implementation of transfer learning for natural language processing. This is a fascinating look at the possibilities of emerging machine learning techniques for transforming the ways that we interact with technology.

Announcements

Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.

When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!

We’ve all been asked to help with an ad-hoc request for data by the sales and marketing team. Then it becomes a critical report that they need updated every week or every day. Then what do you do? Send a CSV via email? Write some Python scripts to automate it? But what about incremental sync, API quotas, error handling, and all of the other details that eat up your time? Today, there is a better way. With Census, just write SQL or plug in your dbt models and start syncing your cloud warehouse to SaaS applications like Salesforce, Marketo, Hubspot, and many more. Go to pythonpodcast.com/census today to get a free 14-day trial.

Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at pythonpodcast.com/hightouch.

Your host as usual is Tobias Macey and today I’m interviewing Paul Azunre about using transfer learning for natural language processing

Interview

Introductions

How did you get introduced to Python?

Can you start by explaining what transfer learning is?

How is transfer learning being applied to natural language processing?

What motivated you to write a book about the application of transfer learning to NLP?

What are some of the applications of NLP that are impractical on intractable without transfer learning?

At a high level, what are the steps for building a new language model via transfer learning?

There have been a number of base models created recently, such as BERT and ERNIE, ELMo, GPT-3, etc. What are the factors that need to be considered when selecting which model to build from?

If there are multiple models that contain the seeds for different aspects of the end goal that you are trying to obtain, what is the feasibility of extracting the relevant capabilities from each of them and combining them in the final model?

What are some of the tools or frameworks that you have found most useful while working with NLP and transfer learning?

How would you characterize the current state of the ecosystem for transfer learning and deep learning techniques applied to NLP problems?

What are the most interesting, innovative, or unexpected applications of transfer learning with NLP that you have seen?

What are the most interesting, unexpected, or challenging lessons that you have learned while working on the book?

When is transfer learning the wrong choice for an NLP project?

What are the trends or techniques that you are most excited for?

Keep In Touch

LinkedIn

Website

@pazunre on Twitter

Picks

Tobias

Infected Mushroom

Paul

Tenet

Closing Announcements

Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management.

Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.

If you’ve learned something or tried out a project from the show then tell us about it! Email [e-mail]) with your story.

To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Join the community in the new Zulip chat workspace at pythonpodcast.com/chat

Links

Transfer Learning for Natural Language Processing by Paul Azunre (affiliate link) Use the code podinit21 at checkout for 35% off all books at Manning!

Low Resource Languages

Fortran

C++

MatLab

MIT 6.003

Transfer Learning

Computer Vision

Deep Neural Network

Convolutional Neural Network (CNN)

Recurrent Neural Network (RNN)

GLUE == General Lanuage Understanding Evaluation

NLP SuperGLUE

NLP Encoder

Named Entity Recognition

ImageNet

Mathematical Optimization

Gradient Descent

Yonder AI

ELMo language model from Allen NLP

Ghana

ArXiv

BERT language model

TF-IDF == Term Frequency – Inverse Document Frequency

Word2Vec

GPT-3

Ghana NLP

Automatic Speech Recognition

ULM Fit

Keras

Tensorflow

Huggingface Transformers

Multi-Task Learning

Fast.ai

OpenAI

AWS SageMaker

Kaggle Kernels

Colab Notebooks

Azure ML Studio

BLEU Score

Khaya application

Android

iOS

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA