If you have ever used Pandas, you know that it is great tool for creating and working with data frames. As with any software, optimizations can be done to make it perform better. Nvidia has developed software called RAPIDS, a data science framework that includes a collection of libraries for executing end-to-end data science pipelines completely on the GPU (www.rapids.ai). Included in RAPIDS is cuDF, which allows data frames to be loaded and manipulated on a GPU. In this post, I am going to discuss some benchmarking I have done with RAPIDS, particularly cuDF. I conducted multiple experiments where I created data frames that ran on the CPU using Pandas and data frames that ran on the GPU using cuDF, then executed common methods on those data frames. I will be using the term “processes” to describe the execution of the methods. I will also be using the convention that Nvidia has used, in which the Pandas data frames are named PDF, for Pandas data frame, and the cuDF data frames are named GDF, for GPU data frame.
For my benchmarking data, I used a CSV file from MIMIC-III, a freely accessible critical care database that was created by the MIT Lab for Computational Physiology (https://mimic.physionet.org/). The file was named MICROBIOLOGYEVENTS.csv. It consisted of 631,726 rows and 16 columns of data. I duplicated the records in the file to create new files that consisted of 5 million, 10 million, 20 million, and 40 million rows and 16 columns of data, respectively. Experiments were then conducted using each of these 5 files. An individual experiment is defined as running a processes on one of the 5 versions of the MICROBIOLOGYEVENTS files as input. Each experiment was repeated 5 times and the results averaged together to give the final results that I list.
The benchmarking was done on both an Nvidia DGX-1 and an IBM POWER Systems AC922 using a single GPU in each. The GPUs in the servers were both Nvidia V100 models, with the DGX-1 GPU having the model with 32GB of RAM and the AC922 having the 16GB model.
For the benchmarking, I ran some common processes on both PDF and GDF data frames and calculated the amount of time it took to run. The processes were done in the following order using a Jupyter Notebook that can be found on my Github (https://github.com/pacejohn/RAPIDS-Benchmarks).
In addition, I created an additional Jupyter Notebook that was used to concatenate 2 data frames. In this experiment, the MICROBIOLOGYEVENTS.csv, which has 631,726 rows, was concatenated onto each of the 5 MICROBIOLOGYEVENTS input files.
In 4 of the 9 experiments, the GDF outperformed the PDF regardless of the input file that was used. In 3 experiments, the PDF outperformed the GDF. Interestingly, in 2 experiments the PDF outperformed the GDF on small data frames but not on the larger ones. In the concatenation experiments, the GDF always outperformed the PDF. The results for the processes that were run on the AC922 are below. The results for the DGX-1 are similar. For complete results, including the actual times for the processes to run and the DGX-1 results, see my Github.
The most remarkable differences in performance were in the following processes.
GDF Outperforms PDF
PDF Outperforms GDF
When the PDF outperformed the GDF, the results were astonishing.
PDF Outperforms GDF on Smaller Data Frames
As shown above, data frames that run on the GPU can often speed up processes that manipulate the data frame by 10x to over 1,000x when compared to data frames that run on the CPU, but this is not always the case. There is also a tradeoff in which smaller data frames perform better on the CPU while larger data frames perform better on the GPU. The syntax for using a GDF is slightly different than using a PDF, but the learning curve is not steep, and the effort is worth the reward. I’m going to try the same benchmarking on some other data sets and use some other methods to see how the results compare. Stay tuned for the next installation.
Follow me on Twitter (@pacejohn) and on LinkedIn (https://www.linkedin.com/in/john-pace-phd-20b87070/). Also, follow my company, Mark III Systems on Twitter (@markiiisystems).
On December 11 and 12, I attended the AI Summit New York City. AI Summit conferences are always top notch and showcase a wide variety of companies that are innovating with their AI technologies. This year was no different. There were companies showing products that ranged from using AI to determine which makeup is best for your skin type to augmenting datasets by creating synthetic data to optimizing data science workflows. The breadth of technologies was definitely interesting.
At my booth, my co-worker, Lisa Stone, and I showed off the latest version my American Sign Language transcription software. The response was even bigger than at other conferences. I had people from major banks, hospital groups, and universities who discussed possible use cases for their industry. A total of 25 people allowed me to gather more training data by filming them signing the alphabet as well as the words “hello”, “good”, “bad”, “I love you”, “you”, “me”, and “your”. Adding words that have motions is the next step of the project. (Spoiler alert: I have that version done for some words, but it needs a little more testing before it moves to prime time.) One exciting experience was when a student from a university in New Jersey came to see the demo. He was interested and we had a good discussion. Later that day, he brought his entire team of 10 developers, data scientists, other AI practitioners to see the demo. You can imagine how surprised I was to see 10 people all show up at once because one man was so excited and wanted them to see it. Finally, I was interviewed on camera by the organizers of the conference because they thought the project was highly innovative and can help make people’s lives better. I call it “AI for Good”.
Just as exciting, I had AI practitioners who discussed the technology with me from a very technical perspective. For me, that is the beauty of these types of conferences. I am able to plunge deeply into the technology and see how other people are doing AI. This information can be used to improve my product. The sharing of ideas is how industries are advanced. As the saying goes “All of us is better than one of us.” How many times do we get to talk to people about the hyperparameters we are using when training our models and how we can optimize them? Not everyone is interested in that, but it’s the type of stuff that gets me excited. We also talked about the hardware I am running the training on. Let’s just say the hardware is quite impressive!
One final note, while in NYC, I got to visit Pace University. No, I’m not the founder and I don’t think I am directly related, but how many people have a university that has their name?
I am looking forward to the conference next year. If you would like to discuss more, let me know. Be sure to follow me on Twitter (@pacejohn), on LinkedIn (https://www.linkedin.com/in/john-pace-phd-20b87070/), and follow my company, Mark III Systems, on Twitter as well (@markiiisystems).
I attended the Super Computing 2019 (SC19) conference in Denver. SC is a very large conference that highlights some of the most cutting-edge work in high-performance computing (HPC), AI, computational fluid dynamics, and other fields where massive computation is required. Groups like NASA, the National Laboratories, including Sandia, Oak Ridge, and Lawrence Livermore, and major research universities all highlighted their work. For someone who loves to see how far computing can be pushed, this is the place to be.
At the conference, I presented my software that uses AI to transcribe American Sign Language (ASL) in real-time as a person signs in front of a camera. As letters of the alphabet are signed, they are written across the screen so they can be read. The ultimate goal of the project is to develop both phone and web apps that allow someone who does not speak ASL to be able to understand what someone who does speak ASL is signing.
Since the project started in August of this year, I have been able to make significant progress. One of the main things I need, and that is needed in any computer vision project, is lots and lots of training data. At several other conferences this year I have asked people to allow me to take a video of them performing the ASL alphabet. I was able to get video of 31 new people performing the alphabet.
At SC19, I added a new twist to the video collection. I had a screen that played a 10-minute loop of various backgrounds, including random colors, a person walking through a castle and a forest, concert video footage, and highlights from the 2019 World Series (sorry Astros fans). People stood in front of the screen as they performed the alphabet. The reason for the screen is very simple. When someone signs, they will rarely be standing in front of a solid, non-moving background. Instead, they will be in a classroom or restaurant or outside or somewhere else where the environment is not static. For the AI software to be generalizable, the training must be done using myriad backgrounds. By adding 10-minutes of different backgrounds, I was able to ensure that each letter that was signed would be on a different background. As I did at previous conferences, I made every attempt to get people with varying hand shapes, colors, sizes, fingernails (both painted and unpainted), and jewelry to sign the alphabet. This will also make the models generalizable to the maximum number of people and reduce bias as much as possible.
As at other conference, the response to the software was very good. I had many, many people come by and try the demo. In fact, I had my first deaf person try it. Honestly, I was quite nervous when she tried. I have been careful to make it as useful as possible, but I wouldn’t know for sure how successful it is until a deaf person tried it. She was very impressed and said it will be very helpful for the deaf community. She even asked how she can help with the development. I am very much looking forward to working with her.
Finally, I was able to do 2 oral presentations of the challenges I face in developing the software. I had several people ask me questions and we had good discussions about some ways to overcome some of the challenges.
If you haven’t had a chance, look at my Twitter feed (@pacejohn) to see the posts I made during the conference and to stay up to date on my latest research. Follow my company on Twitter as well (@markiiisystems) and www.markiiisys.com.
The Nvidia GTC conferences – my favorite times of the year! Some of the greatest minds in AI come together to show off the work they are doing on Nvidia GPUs or the research into AI that they are conducting. For an AI fanatic, it’s a brain overload!
In November, I attend the GTC conference in Washington, DC. Though this conference is significantly smaller than its sister conference in San Jose, it was still packed full of fantastic presentations. The main terms I heard over and over were BERT and PyTorch. BERT is taking the NLP world by storm and PyTorch is quickly gaining momentum and market share among AI/ML/DL frameworks. In this post, I’m going to highlight one of the sessions that stood out to me. In fact, it was possibly the most meaningful and even life-changing presentation I have ever seen at a conference.
Robert Pless, Department Chair & Patrick & Donna Martin Professor of Computer Science at George Washington University, presented a talk entitled “Explainable Deep Learning to Fight Sex Trafficking.” In the talk, he discussed a project he is conducting with the FBI and that is currently in use by the National Center for Missing and Exploited. The project is very simple and elegant, and is having a major impact in helping to stop or prevent crimes against children.
The goal of the project is to be able to identify, down to the specific hotel room, locations where sex trafficking is happening. His team has developed an app called “TraffickCam.” With the app, you can take 4 pictures of a hotel room that are then stored in an image database. These images can then be compared to images that law enforcement officials gather from the Internet or other sources in their trafficking investigations. By identifying distinctive regions that occur in both images, they can identify the specific room where the photograph was taken. From this information, law enforcement can target their investigation much more specifically. As of the presentation time, there are ~356,000 images from ~32,000 hotels in the database and this number is growing. However, this dataset is only able to identify the correct hotel room in 8-35% of the searches. How can this number increase -- more data! I have already added images from 3 hotel rooms since I heard the talk. You can help increase the number, too.
While this topic is quite disturbing, I am incredibly excited to see AI being used in such a helpful and life-changing manner. AI sometimes gets a bad rap, particularly about privacy, but this project is proof that it can change the world for the better. If you haven’t already, please download the app from the App Store or Google Play. It takes less than one minute to take the photos of your hotel room. This one minute can change someone’s life forever. For more information on the project, you can click here or visit their Facebook page. Be sure to watch the YouTube video for more info on how the project started https://www.youtube.com/watch?v=zhfHOR6yc98 and read the paper at https://arxiv.org/abs/1901.11397.
Follow me on Twitter (@pacejohn) for more updates! Also, be sure to follow my company, Mark III Systems (@markiiisystems) and watch their blog at https://www.markiiisys.com/blog/
On November 8-10th, I was able to compete in The University of Texas Southwestern Medical School U-Hack Med Hackathon. I attended the event last year and helped multiple groups as a facilitator and mentor, but this year I applied to compete on a team and was accepted! The 48-hour event was a great learning experience and an amazing way to meet new people from varied backgrounds.
I was on Team 9. Our team’s project was entitled “REAL-TIME CARDIAC ASSESSMENT OF CATHETERIZATION-DERIVED FICK AND CMR-DERIVED FLOW” and focused on using machine learning and AI to reduce the time it takes for children to undergo pediatric heart catherization procedures. For a full description of the project, click here. In the procedure, a wire with a balloon attached to the end is inserted into a large vein near the child’s groin, then guided up through the torso into the heart and surrounding veins and arteries. In the past, this procedure has been done using x-rays to allow the physician to see the wire as it moves. However, this makes an already risky procedure even more dangerous. Studies have shown that children that are exposed to 2-4 hours of x-rays (the amount of time a procedure typically takes) are 3-4 times more likely to develop cancer later in life. More recently, cardiologists have been performing radiation-free heart catheterizations using an MRI to view the wire and balloon. In both procedures, the child has to be put under general anesthesia for the entire time. This much anesthesia carries a high risk of causing developmental and neurological problems. Thus, anything that can be done to reduce the exposure to x-rays and amount of time the child is under anesthesia can significantly reduce potential risks. This was our challenge. Click here for more info on radiation-free heart catheterizations at Children’s Medical Center in Dallas.
Dr. Youssef Arar, a pediatric cardiology fellow at Children’s Medical Center in Dallas was our team lead. Dr. Daniel Castellanos, a pediatric cardiology fellow and colleague of Dr. Arar, was our co-lead. they routinely perform these procedures so they are well acquainted with the intricacies, risks, and ways the procedures can possibly be improved. Dr. Arar presented us with two different tasks that could help reduce the time heart cath procedures take.
The first challenge involved automating calculations of blood flow and pressures. Currently, the physician doing the procedure has to tell a technician what values he is getting. The technician then enters those values into a spreadsheet and has to manually calculate the blood flow and pressure. This is a time consuming and error prone process. Every moment calculations are being performed manually is another moment the child is exposed to radiation or under anesthesia. When the procedure is done in the MRI room, there is a tremendous amount of noise, so communication is impaired significantly. This makes the process even more error prone. Our team developed a web-based application to automate the calculations. To use the app, the values are relayed to the technician, or the technician views them on a screen. As they are input, blood flow, pressure, and other values are automatically calculated, giving the physician immediate answers. Dr. Arar said this app could potentially shorten procedures by 40 minutes and give more accurate calculations!
The second challenge involved visualizing the balloon in the heart during the MRI procedure. An MRI takes an image of a single plane at any given time. If the balloon is not in that plane, the cardiologist cannot see it, so any movement of the balloon is done blindly. This results in the patient having to potentially be repositioned, lengthening the anesthesia time. Dr. Arar asked us to find a way to track the amount of time the balloon is seen during a procedure. By knowing this information, the cardiologists can evaluate the procedure after it is finished to discuss what could have been done differently to keep the balloon in view longer. By determining what adjustments can be made, future procedures can be more efficient, shorter, and safer. Our team used a process called “thresholding” to segment the balloon on the MRI image and count the number of frames it appeared in. This allowed us to calculate the percentage of time the balloon was seen.
Overall, I had a fantastic time. My team was small and we had varied backgrounds. We had a web developer, a computational scientist, a post-doctoral researcher, a PhD student, two MDs, and myself. In the end, our team ended up winning the Lyda Hill Best Overall Team award for our work. Great job everyone!
Don't forget to follow me on Twitter @pacejohn and my company @markiiisystems (https://www.markiiisys.com/blog/)
Yesterday I taught a computer vision workshop. In the section about image classification, I had the attendees train a deep learning model that could correctly classify images of the letters A, B, C, E, and H. To build the model, they started with another previously trained model. This is known as "transfer learning." More specifically, transfer learning is a method where a model developed for a task is reused as the starting point for a model on a second task. It is an optimization that allows rapid progress or improved performance when modeling the second task.
On October 14 and 15, I attended the annual Rice University Data Science Conference. There were a few things I took from the opening days sessions.
The keynote speaker was from a major cancer center in Houston. A large portion of his talk was about data governance and who actually owns a patient’s medical data? Is it the patient, the hospital, the doctor, the insurance company, a combination of these, or none of these? He displayed a map of the United States that showed how this varies from state to state. What was clear is that there is no consistency. This question has to be answered before real analytic work can be done on patient data. This is an absolutely critical problem since data science can improve patient care as well as operational efficiency and profitability of medical institutions, just to name a few. In most instances, only hospital staff have access to data, yet it is sitting unused. I asked the speaker why medical institutions are not making major investments in data science like companies such as Walmart, Wells Fargo, and Uber are making. He stated it is all about outcomes.
The second speaker discussed image analysis of tissue samples. She mentioned that images must be labeled by trained pathologists so there is a ground truth to compare the deep learning model predictions to. Unfortunately, this problem is a catch-22. Pathologists do not have enough time to label images since it is time and labor intensive. Without labeled images, deep learning cannot be used to find areas of interest on the slides. If the images were labeled, the pathologists would have more time work on more complicated cases that deep learning cannot currently solve. That’s where the catch-22 comes in. There is no time to label images due to lack of time, yet labeled images will save time ultimately.
Another hot topic was explainability of deep learning models and how physicians are reticent to adopt AI when no one can tell them why it made the predictions it did. This is where the tradeoff between classical machine learning and deep learning comes in. Maybe the solution to this problem is to use classical machine learning techniques, such as random forests or support vector machines for specific problems, rather than neural networks. In contrast to neural networks, classical machine learning techniques produce models that are explainable. Maybe it is better to start analysis using classical machine learning rather than deep learning and only implement deep learning models when the results are significantly better.
Finally, a speaker discussed the carbon footprint generated by servers that are executing deep learning jobs using GPUs. I had this considered before, but it is a problem that must be addressed since the use of deep learning is becoming more prevalent.
On Day 2, I attended several sessions on data science in healthcare. One session that stood out was about using quantitative measurements to determine the progression of Parkinson’s Disease in patients. Currently, the progression is measured be a neurologist looking at a patient’s movements. At a later time, the neurologist has the patient perform the same movements and makes a qualitative decision about the progression. This is highly subjective. One group is using 2 different methods to measure the progression. One involves hold time for keys on a keyboard. As the disease progresses, there is a marked and statistically significant difference in hold time on keys. They are also using metrics from swipes on mobile devices. As with hold time, swipe dynamics change over time. The pressure applied, the length of the swipe, how straight it is – all of these change. By using these measurements, physicians are able to get clear numerical values that show the progression.
Day 2 was also when the posters were presented. Kate Weeks, a student in the University of the Pacific’s Master’s degree in Data Science, presented her work that she has done on pothole detection using accelerometer data with San Joaquin County in their road maintenance. I have had the privilege of working with her on the project.
If you would like more details on the conference, feel free to reach out. Be sure to follow me on Twitter (@pacejohn) and my company (@markiiisystems).
I had the pleasure of attending the 2019 AI Summit in San Francisco on September 25 and 26.
At the conference, I was able to demo new AI software I am working on. The software is an American Sign Language translator. The goal is to produce an app that allows someone to point their phone camera at a person who is speaking in American Sign Language. The sign language is then transcribed onto the screen. Currently, the software can recognize and transcribe the ASL alphabet. In order to train the deep learning model, I need many, many images of people performing the ASL alphabet. As I explained the work and showed people the demo, I asked if they would let me take a video of their hand as they signed the letters. I had a total of 37 people who let me video them! I was tremendously grateful. The image labeling is being done using IBM Power AI Vision software and the models are being trained using the POWER Systems optimized version of TensorFlow included in the Watson Machine Learning Community Edition software running on an IBM POWER9 AC922 server. I also had the opportunity to do an oral presentation 4 times in the booth to describe my project. The turnout for the presentations was good and I people asked lots of good questions.
The ASL software project is challenging in that combines several machine learning/deep learning techniques that must work together both in concert and independently. The project uses convolutional neural networks (CNN) for object recognition, which is a subset of computer vision, as well as recurrent neural networks (LSTM), and natural language processing (NLP). At a high level, the software must recognize hands, arms, and facial expressions since all are involved in ASL. Upon recognition, it must determine what the hand and arm doing. Is it making a letter, a number, or some other symbol? When is the hand actually making a letter or signing a word and when it is transitioning? The transitions need to be discarded. Combining all of these into a workflow that can run on a mobile app is something that has not been done before but will make a huge impact for the deaf community.
There were several things I found interesting at the conference. Typically, at an AI conference, there is one technology that seems to be everywhere, such as autonomous vehicles or inferencing platforms. However, this conference was different in that no one technology was overrepresented. There were probably more companies that provided image labeling services than any others, but they were not overwhelming.
Overall, I thought the conference was well organized, had a nice breadth of vendors and technologies represented, and was quite productive. I will also be attending the AI Summit in New York in December where we will have an even bigger presence in the booth. Stop by and see the newest version of the ASL demo! If you would like more details on the conference, feel free to reach out. Be sure to follow me on Twitter (@pacejohn) and my company (@markiiisystems).
Over the past few weeks, I have been doing some benchmark testing between the IBM POWER9 AC922 server and the Nvidia DGX-1 server using time series data. The AC922 is IBM's Power processor-based server that is optimized for machine and deep learning. Nvidia's DGX-1 is Nvidia's Intel processor-based server that is optimized for machine and deep learning. Both servers have the latest Nvidia V100 GPU. The AC922 has 2x 16GB V100 GPUs, the DGX-1 has 8x 32GB. This post summarizes the general process used in the benchmarking as well as the results. I have intentionally kept the post somewhat conceptual to illustrate our methodology and key discoveries at a high level. If you want more technical detail, please contact me (email@example.com).
I chose time series data for the benchmark testing for 2 reasons. First, it is something our customers are asking about. Second, time series problems can be investigated using both classical machine learning algorithms like ARIMA, as well as by deep learning using recurrent neural networks, particularly LSTMs. ARIMA runs solely on the CPU whereas training for LSTMs takes place on a GPU. Thus, this testing allowed comparisons of both CPU- and GPU- based processes on both servers. In addition, it allowed me to compare the relative quality of the predictions made by two very distinct techniques.
In this work, I used synthetic data generated using software by Jinyu Xie called Simglucose v0.2.1, "a Type-1 Diabetes simulator implemented in Python for Reinforcement Learning purpose." Predicting blood glucose levels for patients with diabetes is of particular importance because the models can be used in insulin pumps to predict the correct amount of insulin to give to normalize blood glucose levels. The software generates synthetic blood glucose data for children, adolescents, and adults at designated time points. I used data at 1 hour time points. The training data consisted of 2,160 time points (90 days). The prediction data consisted of 336 time points (14 days). So, the models were trained using the 90 days of hourly time points, then tried to predict the hourly values for the next 14 days.
Two different algorithms were used to make the predictions. The first was a classical machine learning technique known as ARIMA (Autoregressive Integrated Moving Average), more specifically, SARIMA (Seasonal Autoregressive Integrated Moving Average). SARIMA is a specialized version of ARIMA that takes into account seasonality of the data (blood glucose levels have a very clear seasonal pattern). The second was an implementation of a recurrent neural network known as an LSTM (Long Short-Term Memory). To perform the comparisons, I used Jupyter notebooks that ran Python scripts. Thanks to Jason Brownlee of Machine Learning Mastery for his publicly available code that was adapted for this project. I compared the predictions for the SARIMA and LSTMs using Mean Squared Error as the quantitative metric of how well the model made predictions. I also calculated the time it took for the training and predictions to be done. For SARIMA, I only calculated the total run time for the training/prediction calculations since there is no distinct prediction phase. For the LSTM, I calculated separate run times for the training and prediction phases since they are distinct processes.
For training and predictions, I used synthetic data for adolescents that I termed as Patients 001, 002, 003, 004, and 005. For each patient, the 2,160 hourly training data points and 336 prediction data points were generated. I then did pairwise comparisons of all patients. For example, I would train a model using the training data for patient 001. I would then use that model to make predictions for patients 001, 002, 003, 004, and 005. The process was repeated for all patients. This gave a total of 25 comparisons. Full pairwise comparisons were performed to evaluate if either SARIMA or LSTM could create models that could be generalized to other patients.
Not surprisingly, the models could not be generalized, which was the expected result. This result underscores the importance of using relevant data to train machine learning models.
Below are examples of how the models performed on predictions. The blue lines are the actual values and the red lines are the predicted values. In the first 2 images, the model was trained on the data for Patient 002. Predictions were then made for Patient 002. In the last 2 images, the model was again trained on the data for Patient 002. Predictions were then made for Patient 001. As you can see, the results were significantly less accurate than when values for Patient 002 were predicted.
The results I obtained were very interesting. Here is a very high-level overview.
So, the burning question is, "Which server is better for time series analysis? The Nvidia DGX-1 or the IBM POWER9 AC922?" The answer is both. For SARIMA and LSTM training, the DGX-1 outperformed the AC922. For LSTM predictions, the AC922 outperformed the DGX-1. Which one should you choose? The answer depends on the use case. If you want to use SARIMA or if LSTM training time is your driving factor, the DGX-1 may be the better choice. If prediction speed is critical, the AC922 could be better.
Admittedly, there are a couple of caveats that must be considered. There were some version differences of pandas, NumPy, TensorFlow, and Keras due to the differing processor architectures. I tried multiple versions of each and the results were very similar. Also, the DGX-1 uses Docker containers while the AC922 uses conda environments. This could lead to some differences as well. Overall, I think these differences have very little effect on the overall benchmark outcomes, but it is something I plan to investigate further. Finally, the models were trained on a small dataset, only 2,160 data points. I will be trying a much larger dataset in the future as well as trying different combinations of hyperparameters to improve forecast accuracy.
The Jupyter notebooks, scripts, and data files, along with all of the summary statistics are available on my GitHub page (https://github.com/pacejohn/Glucose-Level-Prediction).
Brownlee, Jason. 11 Classical Time Series Forecasting Methods in Python (Cheat Sheet). (January 9, 2017) [Online]. Available:
https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/ (Accessed 2019)
Brownlee, Jason. How to Develop LSTM Models for Time Series Forecasting. (November 14, 2018) [Online]. Available:
https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/ (Accessed 2019)
Jinyu Xie. Simglucose v0.2.1 (2018) [Online]. Available: https://github.com/jxx123/simglucose. (Accessed 2019).