 Images from https://www.med.unc.edu/marsicolunginstitute/directory/camille-ehre-phd/  Dr. Camille Ehre, PhD On September 3, 2020, Dr. Camille Ehre, Assistant Professor at the Marsico Lung Institute at the University of North Carolina School of Medicine, published some beautiful and fascinating scanning electron microscope images of the SARS-CoV-2 virus infecting human bronchial epithelial cells. Colored images can be found on Dr. Ehre's faculty page. The work was published in the prestigious New England Journal of Medicine. It's amazing that this seemingly innocuous little particle (in red in the images) could be responsible for so much sickness and death. Thanks for such great work!  Original images from the publication. https://www.nejm.org/doi/full/10.1056/NEJMicm2023328  Links #coronaravirus #sarscov2 #unc #nejm #marsico #sem #microscopy #lungs #virus #infection #science #medicine #biology #disease
0 Comments
 Image from https://arxiv.org/pdf/1707.07734.pdf For the last few weeks, my company, Mark III Systems, along with Vanderbilt University Medical Center Department of Radiology, Dell/EMC, and Nvidia, have been hosting a series of webinars about how AI can augment the great work that radiologists are already doing. AI is never going to replace radiologists, but it does have the potential to make their work a little easier so they can focus on the cases that need their full expertise and experience. The goal of the series is to highlight what is available and what is coming to help achieve this goal.  I had the privilege of being the first presenter. My presentation was entitled "The Greatest Thing to Happen to Computing in 10 Years." I discussed the role of GPUs in AI and how they have made things possible that were not possible just 10 years ago. I then described specific use cases for AI in Radiology. The link to the presentation is below. Enjoy! If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems. #ai #artificialintelligence #machinelearning #radiology #vumc #gpu #dell #emc #nvidia  https://keras.io/ This post is part 3 of my 4 part series on Keras Callbacks. A callback is an object that can perform actions at various stages of training and is specified when the model is trained using model.fit(). In today's post, I give a very brief overview of the Tensorboard callback.  Graphs showing training and validation accuracy and loss. Image from https://github.com/shekkizh/FCN.tensorflow/issues/50  If you are reading this post, then chances are that you have heard of Tensorboard. It is a great visualization tool that works with Tensorflow to allow you to visualize what is going on in the background while your job is running. You can see nice graphs of loss and accuracy, as well as visualizations of your neural network. It has other features as well, but I'll only mention these for now. Take a few minutes to explore it. It is worth the time!  Visualization of neural network While there are multiple arguments that the Tensorboard callback can take, the one I want to discuss today is the log_dir. log_dir allows you to specify the directory where the Tensorboard log files are saved. The files in this directory are read by Tensorboard to create the visualizations. This comes in very handy when you are train with different models or with different hyperparameters. Simply specify different directories and you are all set. If the directory does not exist, it will be created when the callback is used. Here is a short snippet of code that creates and uses the callback. Since this particular job is doing linear regression with 100,000 datapoints, I add that to the directory name. The newly created directory looks something like this. logs/linear_regression_100k_datapoints-20200911-130022/ Now we have the Tensorboard log files being written to a very specific directory that clearly tracks the specific training job. This is a great way to keep up with what you have done! If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems. #artificialintelligence #ai #machinelearning #ml #tensorflow #keras #neuralnetworks #deeplearning #tensorboard #hyperparameters #callbacks #visualization I signed up for Ironman Coeur d'Alene, Idaho, a couple of days ago. Ironman #7 - June 27, 2021. Please COVID.....let us get back to normal by then. A family vacation to the great Pacific Northwest will be fun! #ironman #Triathlon   https://keras.io/ This post is part 2 of my 4 part series on Keras Callbacks. As I stated in part 1, a callback is an object that can perform actions at various stages of training and is specified when the model is trained using model.fit(). They are super helpful and save a lot of lines of code. That's part of the beauty of Keras in general - lots of lines of code saved! Some other callbacks include EarlyStopping, discussed in Part 1 of this series, TensorBoard, and LearningRateScheduler. I'll discuss TensorBoard and LearningRateScheduler in the last 2 parts. For the full list of callbacks, see the Keras Callbacks API documentation. In this post, I will discuss ModelCheckpoint. The purpose of the ModelCheckpoint callback is exactly what it sounds like. It saves the Keras model, or just the model weights, at some frequency that you determine. Want the model saved every epoch? No problem. Maybe after a certain number of batches? Again, no problem. Want to only keep the best model based on accuracy or loss values? You guessed it - no problem. As with other callbacks, you need to define how you want ModelCheckpoint to work and pass those values to model.fit() when you train the model. I'm sure you can read the full documentation on your own, so I'll just give an example from my own work and hit the high points. Here is some example code. Let's discuss the options above.
Here is the output from some training I did using the ModelCheckpoint I defined above. In epochs 3 and 4, the loss decreased, so the model weights were saved. In epochs 5 and 6, the loss did not decreased, so the weights were not saved. Now that we have our weights saved, we can later go back and load them for inference. I won't cover that (although it is simple) for the sake of brevity. Jason Brownlee (@TeachTheMachine) of Machine Learning Mastery has a very good tutorial on how to do it. He always writes great stuff! If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems. #artificialintelligence #ai #machinelearning #ml #tensorflow #keras #neuralnetworks #deeplearning #modelcheckpoint #hyperparameters #callbacks  This post is part 1 of a 4 part series on Keras Callbacks. Training neural networks is an iterative process that can be very time consuming. Rarely do you get the optimal model on your first training run. You try a set of hyperparameters then evaluate how well the model performs. Then you do it again with different hyperparameters. Rinse and repeat, often by using grid search.  One of the great features of Tensorflow 2.x is an API called Keras. It takes the potentially massive amounts of code that are needed to build neural networks and wraps it into a nice, discrete interface. For example, to build an artificial neural network with 53 inputs, 2 hidden layers (one with 6 neurons and the other with 12), and 7 output classes, you can use something like the following code. That's it. 4 lines of code! It takes quite a bit more with standard TensorFlow. Back to training. When training, you need to specify how many epochs you want to train for. An epoch is "one pass over the entire dataset, used to separate training into distinct phases." Sometimes you need to run 100 epochs to properly train your model, sometimes 1,000 or more. But what if you could have the training run automatically stop (not run any more epochs of training) when, according to a metric you set, the model is no longer continuing to improve? Keras has a nice class called "EarlyStopping" that does just that. It "stops training when a monitored metric has stopped improving." EarlyStopping falls into a group of objects known as callbacks that are specified when the model is trained using model.fit(). By setting the EarlyStopping value, you tell the model to quit training when a certain metric has not improved for a specified number of epochs. So maybe after 10 epochs of no improvement, stop the training. If you have specified the training to run for 100 epochs and it can stop at 50 epochs due to no improvement, you have saved 50% of the time you would have needed for training. Saving time is always a benefit! Plus, you can help avoid overfitting. Here's how it works. First, you import EarlyStopping from keras.callbacks. Then specify a value for the "patience" hyperparameter. This is how many epochs that must be trained without improvement for the training to stop. Continue by specifying which metric to evaluate after each epoch of training. Finally, tell your model to use the EarlyStopping values when training. It's that easy. Below is code for running 100 epochs, but stopping after the training accuracy does not improve after 10 epochs. Code Editor
Several other parameters can be specified for EarlyStopping, such as min_delta and verbose. Check out the documentation page for details. I'll discuss some of the other callbacks that are available in future posts. If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems. #artificialintelligence #ai #machinelearning #ml #tensorflow #keras #neuralnetworks #deeplearning #earlystopping #hyperparameters #callbacks Machine learning and deep learning are not trivial subjects. The breadth of the material is pretty amazing. How in the world can we wrap our heads around this mass of information? I would like to share my top 6 online video series about ML and DL. All are available on YouTube for free. These are not in any particular order. I hope you find them helpful - I definitely did! Beginners - the places to get acquainted  1. Deep Lizard Deep Lizard's (@deeplizard) "Machine Learning & Deep Learning Fundamentals" course includes 36 videos, all of which are under 15 minutes, with most being much shorter. They give a very good overview with each topic broken down into bite-sized pieces. Great place to start. www.youtube.com/playlist?list=PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU  2. 3Blue1Brown Grant Sanderson's (@3blue1brown) series of videos on "Neural Networks" nicely combines high level overviews, somewhat deeper explanations, and some fun math in his videos. Plus, he uses the word "squishify" in his video and that makes me laugh. https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi  3. Khan Academy How can you not like Khan Academy (@khanacademy)? They have classes on pretty much everything you could ever want to learn, taught by great teachers, in easy to understand language. Their "Machine Learning" course has 160 videos (don't feel like you have to watch them all). Just pick the videos for the specific section you are interested in and enjoy! https://www.youtube.com/watch?v=yDLKJtOVx5c&list=PLD0F06AA0D2E8FFBA Intermediate - go a little deeper and do some math  4. Andrew Ng - Coursera Andrew Ng (@AndrewYNg) is one of the godfathers of AI and Machine Learning. His "Machine Learning" series from Coursera is probably the best intermediate machine learning class I have found. There is plenty of math and background, but it is a nice continuation of what you learned in the DeepLizard, 3blue1brown, and Khan Academy videos. Did I mention that he is former VP & Chief Scientist of Baidu; Co-Chairman and Co-Founder of Coursera; an Adjunct Professor at Stanford University; Founder and CEO of Landing AI; and Founder of deeplearning.ai? Oh, and he also founded and lead the Google "Brain Project." Wow! When he speaks, we should listen. www.youtube.com/watch?v=qeHZOdmJvFU&list=PLZ9qNFMHZ-A4rycgrgOYma6zxF4BZGGPW Advanced - pretty much as deep as you can go short of writing your dissertation  5. CS229 - Machine Learning - Stanford University This is the Stanford University full semester class taught by Andrew Ng and some grad students in Autumn 2018. The class is 20 videos, each around 1 hour and 20 minutes. This is about as deep as you can go into the topic. Well worth the effort. If you are really feeling motivated, check out the syllabus which has the assignments. They are NOT easy! http://cs229.stanford.edu/syllabus-autumn2018.html https://www.youtube.com/watch?v=jGwO_UgTS7I&list=PLoROMvodv4rMiGQp3WXShtMGgzqpfVfbU 6. CS231n - Convolutional Neural Networks for Visual Recognition - Stanford University This is the Stanford University full semester class taught by Fei-Fei Li (@drfeifei), Serena Yeung (@syeung10), and Justin Johnson (@jcjohnss). Fei-Fei Li is world renown for her work in computer vision. Serena and Justin are both masterful in their teaching. The class is 16 videos, each around 1 hour and 20 minutes. This is about as deep as you can go into the topic. Well worth the effort. If you are really feeling motivated, check out the syllabus which has the assignments. They are NOT easy! http://cs231n.stanford.edu/2017/syllabus https://www.youtube.com/watch?v=vT1JzLTH4G4&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems #artificialintelligence #ai #machinelearning #ml #convolutionalneuralnetworks #cnn #computervision #neuralnetworks #learning #stanford #coursera #khanacademy #3blue1brown #deeplizard Hyperparameters for Classifying Images with Convolutional Neural Networks - Part 2 - Batch Size6/17/2020  In this two part series, I discuss what I consider to be two of the most important hyperparameters that are set when training convolutional neural networks (CNNs) for image classification or object detection. These are learning rate and batch size. In this second part, I will discuss batch size. For an overview of CNNs, please see part 1 of this series. What is batch size and why is it important? First, a little background on neural networks. When training a neural network, weights and biases are used to determine the importance of each value in the input. For example, if the input is a 768×1 vector of values, the network needs to figure out the relative importance of each of these separate values as it pertains to the performance of the network. This is largely dependent on the weights between neurons. During training, an input value is multiplied by a weight (and a bias potentially added). The resulting value is passed to the next neuron where the calculated value is the input for an activation function. See the image below for a visual explanation of the math. The important thing to know is that each of these weights is often initially set to random values. When the input values are passed through the network during the forward pass, a loss value is calculated at the end. That loss value is then used during a process known as back propagation. During back prop, the weights are updated in an effort to make the network more accurate in its predictions. How often back propagation occurs is dependent upon batch size. If the batch size is set to 1, then every time an input passes through the network, back prop will occur and weights will be updated. This can be good because updates are occurring often and can potentially make the performance of the network very good due to how often the weights are updated. This can be bad, though, because the weights are being updated after every input passes through the network. Imagine a network that has millions of weights (Inception-v3 has 24 million parameters). Each time back prop occurs, all of these weights have to be updated. Now imagine that happening hundreds of thousands, or even millions of times as inputs are passed through the network. This is very computationally intensive and time consuming. Low batch sizes can produce very accurate results, but at the cost of computing power and time. One way to decrease training time and computational power, is to increase the batch size to say 16, 32, 256, 1024,or higher. When the batch size is greater than 1, something interesting occurs. A number of inputs equal to the batch size are passed through the network. The calculated loss for each of these inputs is then averaged and that value is used to update the weights. So if the batch size is 32, then 32 inputs are passed through the network before back prop occurs. What are the pros and cons of this? With less back prop occurring, computational requirements decrease and training time decreases. However, the loss value is now an average of multiple inputs rather than just the value derived from a single input. Thus, the weights are updated based on an average of several inputs. This can cause the weight updates to be less granular, resulting in an overall decrease in accuracy and potentially a lack of ability to generalize to new data. Comparing the effects of different batch sizes on a dataset Here are the results from some image classification experiments I conducted using Nvidia‘s DIGITS software running on an Nvidia DGX-1. I used the same input images as in Part 1 and the learning rate was the same for each experiment. 75% of the images were used for training, 25% were used for validation. The difference in these experiments was that the batch size was changed. In each Tensorboard image below, as in Part 1, the two things to look at are the training loss values in blue and training accuracy values in red. Yes, I know there is more to it than that when evaluating how a model is performing, but for illustration, this is enough. The batch sizes were 1, 2, 4, and 8, and the base learning rate was 0.00001. We want to see the loss go down and the accuracy to go up.     Summary of the results
 The question now becomes, “Which is more important? Accuracy or training time?” With this very small data set and small number of epochs, the training did not take terribly long. However, imagine that your training data set is 500,000 images and takes 12 hours to train. In this small experiment, by simply changing the batch size from 1 to 2, we were able to decrease the training time by almost 40%. In your scenario, that would decrease the training time from 12 hours to 7 hours 12 minutes. That is a significant decrease. The decrease in training time is not without a trade off, though. Training time decreased, but so did accuracy. If you need high accuracy, then decreasing the batch size may be a problem. If 80% accuracy is acceptable to have a 40% reduction in training time, then simply changing the batch size can do that for you. One thing to remember, though, is that these values are derived from training on a small data set for a small number of epochs. Your results will definitely vary, but will should be similar. That’s where the fun part of experimenting comes in! If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems #artificialintelligence #ai #machinelearning #ml #convolutionalneuralnetwork #cnn #computervision #learningrate #nvidia #digits #tensorflow #tensorboard #hyperparameters #neuralnetworks #batchsize Hyperparameters for Classifying Images with Convolutional Neural Networks - Part 1 - Learning Rate6/12/2020 In this two part series, I am going to discuss what I consider to be two of the most important hyperparameters that are set when training convolutional neural networks (CNNs) for image classification or object detection. These are learning rate and batch size. In this first part, I will discuss learning rate. Basic Overview of Convolutional Neural Networks Let's start with a very basic overview of how CNNs work. When training a CNN for image classification, a labeled image is used as input for the neural network. A lot of processing is done on this input in multiple layers, such as performing convolutions using kernels of different sizes, passing values through activation functions like ReLu, pooling values to downsize the data, and ultimately, classifying the input image into a category. The image below shows the architecture of the famous AlexNet CNN.  AlexNet (Image from https://neurohive.io/en/popular-networks/alexnet-imagenet-classification-with-deep-convolutional-neural-networks/) The goal during training is to have the network learn the correct weights and biases (parameters) to optimize how well the network classifies images. When an image is classified in the last layer of the network, one of two outcomes is possible. It can be classified correctly or incorrectly. A loss function, sometimes called a cost function, is used to calculate a value known as the loss. Typically, the lower the loss, the better the performance. The goal is to optimize the network so the loss is as low as possible without overfitting. There are many types of loss functions, with cross entropy being one of the most common. Neural networks use a process called back propagation to update the weights and biases of the network in order to minimize the loss. The math behind how back prop works is beyond this post, but Matt Mazur (@mhmazur) has an excellent article that clearly explains it. Learning rate is a critical part of back prop. If the learning rate is too high, the weights and biases will be adjusted significantly on each pass. If the learning rate is too low, the adjustments will be very small. Either of these can cause the network to never converge. The art is to find a learning rate that will minimize the loss. A couple of tips to help you find the best learning rate First, you can either use a constant learning rate, or you can decrease the learning rate over time. This is known as decay. For example, for the first 10 epochs, you may use 0.001 for the learning rate. At epoch 11, you drop the learning rate to 0.0001. At epoch 20, it drops to 0.00001. The reason for having the learning rate drop over time is so the adjustments to the weights become less variable as the network learns. It's like getting a haircut. The barber first cuts off long lengths of hair. Then they make some adjustments and cut shorter lengths. Finally, they make very small cuts around your ears for the final polishing. This is exactly what happens when the learning rate drops after a period of time. I recommend using decay. Second, start with a high learning rate, say 0.1, or a low learning rate, such as 1e-7. Learning rates tend to vary from 0.1 to 1e-7, although the large and small ends of that range are much less common. You will most likely find that these do not work well and may give errors very early in training. In your next set of experiments, set the learning rate to 0.01, then try again with 1e-6. Keep working towards a value near the middle and compare the results of each experiment. You can also start with a middle value, say 0.001 or 0.00001 and work toward the upper and lower numbers in the range. Once you have trained your network using different learning rates, evaluate the results. At which learning rate was the accuracy best? This is typically the learning rate you want to use. However, learning rate is not the only hyperparameter that makes a difference in model performance. Next time we will discuss batch size which must also be adjusted to find the optimal value. Examples of the effect on learning rate on results Here are the results from some image classification experiments I conducted using Nvidia's DIGITS software running on an Nvidia DGX-1. The training images (see below for some examples) were letters from tombstones. There were 10 different letters to classify - A, B, C, E, H, N, O, R, S, and T, and a total of 700 images for each letter. 75% of the images were used for training, 25% were used for validation. In each experiment, the learning rate was changed and decay was used to lower the learning rate by 1/10th every 10th epoch.  In each Tensorboard image below, the two things to look at are the training loss values in blue and training accuracy values in red. Yes, I know there is more to it than that when evaluating how a model is performing, but for illustration, this is enough. The batch size was 1 in each experiment. We want to see the loss go down and the accuracy to go up.    Summary of the results
 From this very simple analysis, we would say that a learning rate of 0.0001 is the best choice for training our model on this data set. Does this mean this is the learning rate we should use with every data set? No. Remember, there is no magical formula. It takes trial and error. With this set of images in these experiments, this just happened to be the optimal value. Stay tuned for the next installment where I will show how changing the batch size affected the results. If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems #artificialintelligence #ai #machinelearning #ml #convolutionalneuralnetwork #cnn #computervision #learningrate #nvidia #digits #tensorflow #tensorboard #hyperparameters #neuralnetworks
 Hi, I'm Clippy. Remember me? A few days ago, I was writing a document in Microsoft Word and found something very interesting. I inserted a picture and noticed it automatically created a caption for the image. I was really impressed. Image captioning is not an easy process. You have to combine convolutional neural networks (CNNs) with recurrent neural networks (RNNs), particularly Long Short Term Memory (LSTM) RNNs. Putting those models in production in the standard word processing program is a great example of artificial intelligence in our every day work and how AI is being integrated into all parts of our lives.  Image from https://raw.githubusercontent.com/yunjey/pytorch-tutorial/master/tutorials/03-advanced/image_captioning/png/model.png The image above is a workflow for image captioning. First, you input the image. Next, use a CNN to get a feature vector. Finally, use the feature vector as input for the LSTM. That's all there is to it (too bad each of those steps is not trivial). Here is a fun video I made to demonstrate how the process works in Microsoft Word. Try it out for yourself! If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn, LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/, and follow my company, Mark III Systems, on Twitter @markiiisystems #artificialintelligence #AI #machinelearning #MLDL #microsoftword #neuralnetworks #imagecaptioning #recurrentneuralnetworks #rnn #convolutionalneuralnetworks #cnn #lstm #longshorttermmemory #microsoft #microsoftword #microsoftoffice #office365
|