Machine learning and deep learning are not trivial subjects.  The breadth of the material is pretty amazing.  How in the world can we wrap our heads around this mass of information?  I would like to share my top 6 online video series about ML and DL.  All are available on YouTube for free.  These are not in any particular order.  I hope you find them helpful - I definitely did!

1. Deep Lizard
Deep Lizard's (@deeplizard) "Machine Learning & Deep Learning Fundamentals" course includes 36 videos, all of which are under 15 minutes, with most being much shorter.  They give a very good overview with each topic broken down into bite-sized pieces.  Great place to start.

​​www.youtube.com/playlist?list=PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU

2. 3Blue1Brown
Grant Sanderson's (@3blue1brown) series of videos on "Neural Networks" nicely combines high level overviews, somewhat deeper explanations, and some fun math in his videos.  Plus, he uses the word "squishify" in his video and that makes me laugh.

​https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi

3. Khan Academy
How can you not like Khan Academy (@khanacademy)? They have classes on pretty much everything you could ever want to learn, taught by great teachers, in easy to understand language.  Their "Machine Learning" course has 160 videos (don't feel like you have to watch them all).  Just pick the videos for the specific section you are interested in and enjoy!

https://www.youtube.com/watch?v=yDLKJtOVx5c&list=PLD0F06AA0D2E8FFBA

 4. Andrew Ng - Coursera
Andrew Ng (@AndrewYNg) is one of the godfathers of AI and Machine Learning.  His "Machine Learning" series from Coursera is probably the best intermediate machine learning class I have found.  There is plenty of math and background, but it is a nice continuation of what you learned in the DeepLizard, 3blue1brown, and Khan Academy videos.  Did I mention that he is former VP & Chief Scientist of Baidu; Co-Chairman and Co-Founder of Coursera; an Adjunct Professor at Stanford University; Founder and CEO of Landing AI; and Founder of deeplearning.ai?  Oh, and he also founded and lead the Google "Brain Project."  Wow!  When he speaks, we should listen. ​
​
www.youtube.com/watch?v=qeHZOdmJvFU&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW

5. CS229 - Machine Learning - Stanford University
This is the Stanford University full semester class taught by Andrew Ng and some grad students in Autumn 2018.  The class is 20 videos, each around 1 hour and 20 minutes. This is about as deep as you can go into the topic.  Well worth the effort.  If you are really feeling motivated, check out the syllabus which has the assignments.  They are NOT easy!  http://cs229.stanford.edu/syllabus-autumn2018.html
​
https://www.youtube.com/watch?v=jGwO_UgTS7I&list=PLoROMvodv4rMiGQp3WXShtMGgzqpfVfbU

6. CS231n - Convolutional Neural Networks for Visual Recognition - Stanford University
This is the Stanford University full semester class taught by Fei-Fei Li (@drfeifei), Serena Yeung (@syeung10), and Justin Johnson (@jcjohnss).  Fei-Fei Li is world renown for her work in computer vision.  Serena and Justin are both masterful in their teaching.  The class is 16 videos, each around 1 hour and 20 minutes. This is about as deep as you can go into the topic.  Well worth the effort.  If you are really feeling motivated, check out the syllabus which has the assignments.  They are NOT easy!  http://cs231n.stanford.edu/2017/syllabus

https://www.youtube.com/watch?v=vT1JzLTH4G4&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk

In this two part series, I discuss what I consider to be two of the most important hyperparameters that are set when training convolutional neural networks (CNNs) for image classification or object detection.  These are learning rate and batch size.  In this second part, I will discuss batch size.  For an overview of CNNs, please see part 1 of this series. 

First, a little background on neural networks.  When training a neural network, weights and biases are used to determine the importance of each value in the input. For example, if the input is a 768x1 vector of values, the network needs to figure out the relative importance of each of these separate values as it pertains to the performance of the network.  This is largely dependent on the weights between neurons.  During training, an input value is multiplied by a weight (and a bias potentially added).  The resulting value is passed to the next neuron where the calculated value is the input for an activation function.  See the image below for a visual explanation of the math.  The important thing to know is that each of these weights is often initially set to random values.  When the input values are passed through the network during the forward pass, a loss value is calculated at the end.  That loss value is then used during a process known as back propagation.  During back prop, the weights are updated in an effort to make the network more accurate in its predictions.  How often back propagation occurs is dependent upon batch size.

 If the batch size is set to 1, then every time an input passes through the network, back prop will occur and weights will be updated.  This can be good because updates are occurring often and can potentially make the performance of the network very good due to how often the weights are updated.  This can be bad, though, because the weights are being updated after every input passes through the network.  Imagine a network that has millions of weights (Inception-v3 has 24 million parameters).  Each time back prop occurs, all of these weights have to be updated.  Now imagine that happening hundreds of thousands, or even millions of times as inputs are passed through the network.  This is very computationally intensive and time consuming.  Low batch sizes can produce very accurate results, but at the cost of computing power and time.

One way to decrease training time and computational power, is to increase the batch size to say 16, 32, 256, 1024,or higher.  When the batch size is greater than 1, something interesting occurs.  A number of inputs equal to the batch size are passed through the network.  The calculated loss for each of these inputs is then averaged and that value is used to update the weights.  So if the batch size is 32, then 32 inputs are passed through the network before back prop occurs.  What are the pros and cons of this?  With less back prop occurring, computational requirements decrease and training time decreases.  However, the loss value is now an average of multiple inputs rather than just the value derived from a single input.  Thus, the weights are updated based on an average of several inputs.  This can cause the weight updates to be less granular, resulting in an overall decrease in accuracy and potentially a lack of ability to generalize to new data.

The question now becomes, "Which is more important?  Accuracy or training time?"  With this very small data set and small number of epochs, the training did not take terribly long.  However, imagine that your training data set is 500,000 images and takes 12 hours to train.  In this small experiment, by simply changing the batch size from 1 to 2, we were able to decrease the training time by almost 40%.  In your scenario, that would decrease the training time from 12 hours to 7 hours 12 minutes.  That is a significant decrease.  The decrease in training time is not without a trade off, though.  Training time decreased, but so did accuracy.  If you need high accuracy, then decreasing the batch size may be a problem.  If 80% accuracy is acceptable to have a 40% reduction in training time, then simply changing the batch size can do that for you.  One thing to remember, though, is that these values are derived from training on a small data set for a small number of epochs.  Your results will definitely vary, but will should be similar.  That's where the fun part of experimenting comes in!

In this two part series, I am going to discuss what I consider to be two of the most important hyperparameters that are set when training convolutional neural networks (CNNs) for image classification or object detection.  These are learning rate and batch size.  In this first part, I will discuss learning rate.

Basic Overview of Convolutional Neural Networks

Let's start with a very basic overview of how CNNs work. When training a CNN for image classification, a labeled image is used as input for the neural network.  A lot of processing is done on this input in multiple layers, such as performing convolutions using kernels of different sizes, passing values through activation functions like ReLu, pooling values to downsize the data, and ultimately, classifying the input image into a category.  The image below shows the architecture of the famous AlexNet CNN. 

AlexNet (Image from https://neurohive.io/en/popular-networks/alexnet-imagenet-classification-with-deep-convolutional-neural-networks/)

The goal during training is to have the network learn the correct weights and biases (parameters) to optimize how well the network classifies images.  When an image is classified in the last layer of the network, one of two outcomes is possible.  It can be classified correctly or incorrectly.  A loss function, sometimes called a cost function, is used to calculate a value known as the loss.  Typically, the lower the loss, the better the performance.  The goal is to optimize the network so the loss is as low as possible without overfitting.  There are many types of loss functions, with cross entropy being one of the most common.  ​

Neural networks use a process called back propagation to update the weights and biases of the network in order to minimize the loss.  The math behind how back prop works is beyond this post, but Matt Mazur (@mhmazur) has an excellent article that clearly explains it.  Learning rate is a critical part of back prop.  If the learning rate is too high, the weights and biases will be adjusted significantly on each pass.  If the learning rate is too low, the adjustments will be very small.  Either of these can cause the network to never converge.  The art is to find a learning rate that will minimize the loss.  

Examples of the effect on learning rate on results

Here are the results from some image classification experiments I conducted using Nvidia's DIGITS software running on an Nvidia DGX-1.  The training images (see below for some examples) were letters from tombstones.  There were 10 different letters to classify - A, B, C, E, H, N, O, R, S, and T, and a total of 700 images for each letter.  75% of the images were used for training, 25% were used for validation.  In each experiment, the learning rate was changed and decay was used to lower the learning rate by 1/10th every 10th epoch. ​

In each Tensorboard image below, the two things to look at are the training loss values in blue and training accuracy values in red. Yes, I know there is more to it than that when evaluating how a model is performing, but for illustration, this is enough. The batch size was 1 in each experiment.  We want to see the loss go down and the accuracy to go up. ​

The image above is a Workflow for Image Captioning. Input the image, use CNN to get feature vector, then use feature vector as input for the LSTM. (Image Source here)
​
Click here to watch my demo on YouTube!

If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems

#artificialintelligence #AI #machinelearning #MLDL #microsoftword #neuralnetworks #imagecaptioning #recurrentneuralnetworks #rnn #convolutionalneuralnetworks #cnn #lstm #longshorttermmemory

I recently published a blog on 5 Ways AI is Being Used in the Fight Against COVID-19.  Andy Lin from Mark III Systems interviewed me for their Stackin' It Up series where I gave some more details.  I think you will find the video quite helpful!

People have asked me about how Artificial Intelligence (AI) is being used in the fight against COVID-19.  The answer is that it is being used in myriad of ways!  In this blog, I’ve discussed five innovative AI projects being used to help stop this global pandemic and save lives.

1 - Arguably the most discussed and used dashboard is the “COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University”. You have likely seen it on television press conferences and it was recently featured on NPR. Developed by Ensheng Dong, Hongru Du, and Lauren Gardner at Johns Hopkins University, it provides stunning visuals of real-time data such as “Cumulative Confirmed Cases”, “Active Cases”, and “Testing Rate”, just to name a few.  They have also open-sourced all of the data on Github at https://github.com/CSSEGISandData/COVID-19. They were the first to do this and it was recently the number one repository trending on GitHub!  For technical details, you can read the article in the journal Lancet.

5- Finally, on April 10, Apple and Google put out a press release stating that they are partnering on COVID-19 contact tracing technology that will be released in May. “The new system … would use short-range Bluetooth communications to establish a voluntary contact-tracing network, keeping extensive data on phones that have been in close proximity with each other. Official apps from public health authorities will get access to this data, and users who download them can report if they’ve been diagnosed with COVID-19. The system will also alert people who download them to whether they were in close contact with an infected person.” Let me reiterate - this app will actually alert you if you come into close contact with an infected person.  If this happens, you can immediately take precautions and get tested early.  This could literally be a lifesaver!
​
Here is a list of helpful sites that use AI to help fight COVID-19.  If you know of any others, please leave them in the comments or email them.  Please be sure to keep up social distancing.  Nothing is more effective to stop the spread!

The How We Feel Project - https://howwefeel.org/
Health Map - https://www.healthmap.org/covid-19/?mod=article_inline
BBC Coronavirus Tracking - https://www.bbc.com/news/world-51235105
Coronavirus Dashboard by thebaselab - https://coronavirus.thebaselab.com/
Genomic epidemiology of novel coronavirus by NextStrain - https://nextstrain.org/ncov/global
COVID-19 tracker for Singapore - https://www.againstcovid19.com/singapore/dashboard
Covid-19 tracker from Microsoft - https://www.bing.com/covid
IHME COVID-19 Projections - http://www.healthdata.org/covid
Wim Naude’s excellent review of AI technologies involved in the fight against COVID-19 - https://towardsdatascience.com/artificial-intelligence-against-covid-19-an-early-review-92a8360edaba

If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems

#artificialintelligence #AI #machinelearning #MLDL #COVID19 #JHU #johnshopkinsuniversity #HowWeFeelProject #HowWeFeel #apple #IHME

Data science, at its core, is about time. Time to increasing sales, time to finding cures for diseases, time to understanding.  Thus, it is critical to think about what hardware you are using for deep learning training, which is by far the most computationally intensive part of artificial intelligence and data science in general. A 10% reduction in training time can make a huge difference in employee productivity and time to results.  How about a 25% decrease in training time?  Let me show you how to get it!


I recently read a Medium article by Thilina Rajapakse comparing 7 different BERT models. BERT is the current state-of-the-art training algorithm for natural language processing (NLP).  In his article, he compared the models in terms of final accuracy and training time.  His work was done using a custom built, AMD processor-based workstation with an Nvidia Titan RTX GPU, built upon the Turing architecture.  This is a very popular PC/workstation GPU.  In my conversations with customers, I often hear that they do their training on their laptop or a workstation with a configuration similar to that used by Thilina.  His article, along with these customer comments, motivated me to determine how great of a difference an enterprise level server with top-shelf CPUs and GPUs can make in training time for these same models.  The work I am presenting here did not focus on how the models work or comparing accuracy, but rather on training time.  Since the models were so different, they allowed for a broad, unbiased comparison.


I set up two different enterprise servers in our lab which I will call Server 1 (S1) and Server 2 (S2), to run the same training that Thilina did. My goal was to compare training times between S1, S2, and Thilina’s workstation (WS).  Whereas WS had a Titan RTX GPU, S1 and S2 had Nvidia V100 GPUs, Nvidia’s most powerful GPU.  I expected S1 and S2 to outperform WS, but I did not know how much difference there would be.  Would it be significant?  Would it justify the difference in price?  Those were the two main questions I was trying to answer.  In essence, by running deep learning training on an enterprise server, how much more productive, and thus profitable, could a company be?


To keep this post concise, only the summary of the results is presented (see Table 1 below).  Full details of the training performed and the specs of S1, S2, and WS can be found on my Github page.  In Table 1, each model is listed, followed by the training time on that model for S1, S2, and WS.  Overall, S1 reduced training time compared S2 and WS by 13.6% and 24.7%, respectively.  S2 reduced training time compared to WS by 16.6%.  An example from a customer is even more dramatic.  A data scientist took a random forest model he was training on his laptop and trained it on a server very similar to S1.  The training time decreased from 152 minutes to 26 minutes.  That is 6x speedup.

Customers often ask me about how to operationalize the machine learning models they have created.  It’s common that they have the model, but are struggling with how to apply it to new or existing business data sets. Another line of questioning is about “How do we move the machine learning model forward into production?”

I wrote this article to provide readers an overview of the basic steps of operationalizing machine learning models.

 Creating a logistic regression model that can be operationalized 
 
Believe it or not, you simply create the model and save it to a file.  This is typically done in a script that is separate from the script that will be used in production.  pickle and joblib are common ways to save random forest, logistic regression, or other classical machine learning models. In this post, I will show the process for pickle.  joblib is very similar.  For neural networks created with Keras, it is recommended to use keras.models.  Once the file is created, you later load it and use it to score your data.
 
Create the model file and save it

1. Create (fit) your model
2. Save it to a file.

 
Import the model from the file and use it on new data

1. Import (load) your model file
2. Get results using the loaded model

 
Example:  Logistic Regression
Script to create and save model
import pickle
# Import other modules such as scikit-learn to create your model
 
# Load and manipulate data as necessary
 
#Create model and fit
model = LogisticRegression()
model.fit(X_train, Y_train)
 
# Save the model to a file.  Common extensions for the file are .h5, .pkl, or .sav.  It does not matter what extension you use.
filename = “my_model_file.h5”
pickle.dump(model, open(filename, ‘wb’))
 
Script used in production
 import pickle
# Import other modules such as scikit-learn to be able to use your model
 
# Load and manipulate data as necessary
 
# Load (open) the saved model file and score
filename = “my_model_file.h5”
loaded_model = pickle.load(open(filename, ‘rb’))
result = loaded_model.score(X_test, Y_test))
 
# Evaluate your results as needed for your situation
 
That’s all there is to it!  For more in-depth instructions on how to operationalize models using joblib from scikit-learn or saving and loading models in Keras, check out the following helpful websites:
 
Jason Brownlee’s Machine Learning Mastery website
https://machinelearningmastery.com/save-load-machine-learning-models-python-scikit-learn/
 
Kaggle
https://www.kaggle.com/prmohanty/python-how-to-save-and-load-ml-models
 
keras.io
https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model

Do you work with clients who need help with operationalizing machine learning models? I’d love to hear what questions you commonly come across, please share by adding a comment below.
 
If you have questions and want to connect, you can message me on LinkedIn, Twitter:
Follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems

#ai #machinelearning #MLDL #artificialintelligence #operationalize #production #python #keras #scikit-learn 

Tina Reed and Heather Landi from Fierce Healthcare recently wrote an article discussing the Top 5 things that will be generating buzz at the Healthcare Information and Management Systems Society (HIMSS) 2020 conference. The second thing on their list stated “Natural language processing will be all the rage.”  This is very interesting considering that is exactly what I was going to be speaking about at the conference! My talk was titled “Healthcare, BERT AI, and Next-Gen Natural Language Processing/Natural Language Understanding - How BERT enables institutions to ‘read’ chart notes with unprecedented speed and accuracy.” Unfortunately, HIMSS was cancelled last week due to concerns surrounding COVID-19 (coronavirus). Although I won’t be able to do the presentation at the conference, I will be posting a video of the same presentation shortly.  Stay tuned and see how we are staying at the forefront of AI that is “all the rage.” #NLP #BERT #HIMSS2020 #TinaReed @TreedinDC #HeatherLandi @HeatherLandi #MarkIIISystems @MarkIIISystems #NaturalLanguageProcessing #alltherage https://www.linkedin.com/posts/john-pace-phd_here-are-the-top-5-things-that-will-be-generating-activity-6641744058869960704-gUXJ
​