John Pace
  • Home
  • About
  • Contact

​​
​
Data Scientist, husband, father of 3 great daughters, 5x Ironman triathlon finisher, just a normal guy who spent a lot of time in school.
Let’s explore data science, artificial intelligence, machine learning, and other topics together.

OpenPOWER Summit 2019

8/27/2019

0 Comments

 
Picture

On August 19 and 20, I attended the OpenPOWER Summit in San Diego, CA.  This was my first time to attend the conference, so I was a little unsure of what to expect.  My main purpose in attending was to host a speaking session with Linton Ward, Distinguished Engineer for IBM, and Chris Sullivan, Assistant Director of the Center for Genome Research and Biocomputing at Oregon State University, where we spoke about a group we formed earlier this year in conjunction with Charlie Foretich from Tech Data, called the “OpenPOWER Solution Builder Community” (OSBC).
​
Picture
Linton Ward, Distinguished Engineer, IBM (left), Chris Sullivan, Assistant Director of the Center for Genome Research and Biocomputing at Oregon State University (center), Charlie Foretich, Field Solution Architect for IBM Systems at Tech Data (right)
PictureIBM POWER9 AC922 server

The OSBC is a group of researchers, university faculty, members of industry, value added resellers, systems integrators, and business partners who use and/or sell IBM POWER Systems, particularly the IBM POWER9 AC922.  Our overarching goals are to foster collaboration, share experiences, and give feedback.  As one of the speakers at the conference said, “All of us is better than one of us.”


​
Currently, there is not much out there about what people are doing with POWER Systems, how they are doing it, what challenges they face, or how they have overcome those challenges.  We plan to change that.  In the presentation, Linton provided an overview of the OSBC, Chris talked about it from the hardware perspective, and I discussed the AI side of things, particularly using IBM Watson Machine Learning Community Edition and IBM Visual Insights (previously PowerAI Vision).
​

​The presentation was well-received, and many attendees of the conference were excited about the OSBC.  This is something that is needed and will provide a significant benefit to the community.  To learn more about the community you can visit the OSBC website. You do not have to be a member of the Open Power Foundation to read the forum, but you do have to be a member to post.

Chris Sullivan has put together demos that compare the performance of the IBM POWER9 AC922 versus several different Intel-based servers.  It is remarkable and worth a 10-minute look.  Chris has also put together a couple of rich GitHub repository with scripts and tools specific to the AC922.
Picture

At the conference, a major announcement was made.  IBM is moving the OpenPOWER Foundation to The Linux Foundation and open sourcing the POWER Instruction Set Architecture (ISA). Tech Crunch is just one of the many outlets that gives more details of the announcement found here.  I think this will have a positive impact overall because it will allow chip manufacturers, such as FPGA manufacturer, Xilink, to innovate at a faster speed and create products that have a larger impact.


Finally, I was interviewed for a podcast about the OSBC with Linton and Chris by Luke Schatz (@lukeschantz) from IBM.  It was a great experience and will post the podcast once it is published.  You can also check out my Twitter feed (@pacejohn) to see other highlights from the conference.

​If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn and LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/.

#artificialintelligence #ai #machinelearning #ml #ibm #powersystems #openpower #linux #openpowerfoundation #linuxfoundation #techdata #oregonstateuniversity #AC922 #watsonmachinelearning #watson #OSBC #fpga
0 Comments

AI, Machine Learning, and Time-Series Data:  A Performance Evaluation of Various Scenarios

8/1/2019

0 Comments

 
Picture
Photo by Claudel Rheault on Unsplash

Over the past few weeks, I have been doing some benchmark testing between the IBM POWER9 AC922 server and the Nvidia DGX-1 server using time series data.  The AC922 is IBM's Power processor-based server that is optimized for machine and deep learning.  Nvidia's DGX-1 is Nvidia's Intel processor-based server that is optimized for machine and deep learning.  Both servers have the latest Nvidia V100 GPU.  The AC922 has 2x 16GB V100 GPUs, the DGX-1 has 8x 32GB.  This post summarizes the general process used in the benchmarking as well as the results.  I have intentionally kept the post somewhat conceptual to illustrate our methodology and key discoveries at a high level.
PictureFrom https://www.fool.com/investing/2017/03/10/are-you-nervous-about-the-stock-market-heres-what.aspx

I chose time series data for the benchmark testing for 2 reasons.  First, it is something our customers are asking about.  Second, time series problems can be investigated using both classical machine learning algorithms, like ARIMA, as well as by deep learning using recurrent neural networks, particularly LSTMs.  ARIMA runs solely on the CPU whereas training for LSTMs takes place on a GPU.  Thus, this testing allowed comparisons of both CPU- and GPU-based processes on both servers.  In addition, it allowed me to compare the relative quality of the predictions made by two very distinct techniques.


In this work, I used synthetic data generated using software by Jinyu Xie called Simglucose v0.2.1, "a Type-1 Diabetes simulator implemented in Python for Reinforcement Learning purpose."  Predicting blood glucose levels for patients with diabetes is of particular importance because the models can be used in insulin pumps to predict the correct amount of insulin to give to normalize blood glucose levels. The software generates synthetic blood glucose data for children, adolescents, and adults at designated time points.  I used data at 1 hour time points.  The training data consisted of 2,160 time points (90 days).  The prediction data consisted of 336 time points (14 days).  So, the models were trained using the 90 days of hourly time points, then tried to predict the hourly values for the next 14 days. 

Two different algorithms were used to make the predictions.  The first was a classical machine learning technique known as ARIMA (Autoregressive Integrated Moving Average), more specifically, SARIMA (Seasonal Autoregressive Integrated Moving Average).  SARIMA is a specialized version of ARIMA that takes into account seasonality of the data (blood glucose levels have a very clear seasonal pattern).  The second was an implementation of a recurrent neural network known as an LSTM (Long Short-Term Memory).  To perform the comparisons, I used Jupyter notebooks that ran Python scripts.  Thanks to Jason Brownlee (@TeachTheMachine) of Machine Learning Mastery for his publicly available code that was adapted for this project.  I compared the predictions for the SARIMA and LSTMs using Mean Squared Error as the quantitative metric of how well the model made predictions.  I also calculated the time it took for the training and predictions to be done.  For SARIMA, I only calculated the total run time for the training/prediction calculations since there is no distinct prediction phase.  For the LSTM, I calculated separate run times for the training and prediction phases since they are distinct processes.

For training and predictions, I used synthetic data for adolescents that I termed as Patients 001, 002, 003, 004, and 005.  For each patient, the 2,160 hourly training data points and 336 prediction data points were generated. I then did pairwise comparisons of all patients.  For example, I would train a model using the training data for patient 001.  I would then use that model to make predictions for patients 001, 002, 003, 004, and 005.  The process was repeated for all patients.  This gave a total of 25 comparisons.  Full pairwise comparisons were performed to evaluate if either SARIMA or LSTM could create models that could be generalized to other patients.

Not surprisingly, the models could not be generalized, which was the expected result.  This result underscores the importance of using relevant data to train machine learning models.

Below are examples of how the models performed on predictions.  The blue lines are the actual values and the red lines are the predicted values.  In the first 2 images, the model was trained on the data for Patient 002.  Predictions were then made for Patient 002.  In the last 2 images, the model was again trained on the data for Patient 002.  Predictions were then made for Patient 001.  As you can see, the results were significantly less accurate than when values for Patient 002 were predicted. ​

Figure 1: Predictions for Blood Glucose Levels for Patient 002. The top image shows the predictions using SARIMA. The bottom image shows predictions with the LSTM. Since the predictions are made using the data from the same patient, the predictions should be quite accurate, as is shown in the graphs.
Picture
Picture

Figure 2: Predictions for Blood Glucose Levels for Patient 001. The top image shows the predictions using SARIMA. The bottom image shows predictions with the LSTM. Since the predictions are made using the data for a different patient (Patient 002), the predictions should be less accurate, as is shown in the graphs.
​
Picture
Picture
The results I obtained were very interesting.  Here is a very high-level overview.

  • For the SARIMA training and predictions, which run strictly on the CPU, the DGX-1 performed an average of 42.9% faster than the AC922 (range 29-48% per patient). 
  • For the LSTM training phase, which runs strictly on the GPU, the DGX-1 performed 8.0% faster than the AC922 (range 6-11% per patient).
  • For the LSTM inference phase, which runs strictly on the CPU, the AC922 performed 16.9% faster than the DGX-1 (range 8-28% per patient).
  • The total training/prediction time for SARIMA was several orders of magnitude faster than training and prediction using LSTM.  The average total training/prediction time for SARIMA on the DGX-1 was 7.7 seconds and 13.8 seconds on the AC922.  However, the average LSTM training/prediction time on the DGX-1 was 2,478.1 seconds (321.8 times slower than SARIMA) and 2,693.7 seconds on the AC922 (179.6 times slower than SARIMA). 
  • In some cases, the MSE for SARIMA was lower than for LSTM and vice versa, so it is difficult to say which model actually performed better in making predictions.  In all cases, the MSE for the SARIMA and the LSTM were comparable with each other (see images above).  This outcome could vary significantly based upon how the LSTM network is built and how the hyperparameters for the SARIMA model are chosen.   For these benchmarks, the goal was not to find optimum hyperparameters, but rather to evaluate the performance of one server versus the other on identical machine learning/deep learning tasks.

So, the burning question is, "Which server is better for time series analysis?  The Nvidia DGX-1 or the IBM POWER9 AC922?"  The answer is both.  For SARIMA and LSTM training, the DGX-1 outperformed the AC922.  For LSTM predictions, the AC922 outperformed the DGX-1.  Which one should you choose?  The answer depends on the use case.  If you want to use SARIMA or if LSTM training time is your driving factor, the DGX-1 may be the better choice.  If prediction speed is critical, the AC922 could be better. ​

Admittedly, there are a couple of caveats that must be considered.  There were some version differences of pandas, NumPy, TensorFlow, and Keras due to the differing processor architectures.  I tried multiple versions of each and the results were very similar.  Also, the DGX-1 uses Docker containers while the AC922 uses conda environments.  This could lead to some differences as well.  Overall, I think these differences have very little effect on the overall benchmark outcomes, but it is something I plan to investigate further.  Finally, the models were trained on a small data set, only 2,160 data points.  I will be trying a much larger data set in the future as well as trying different combinations of hyperparameters to improve forecast accuracy.

The Jupyter notebooks, scripts, and data files, along with all of the summary statistics  are available on my GitHub page (https://github.com/pacejohn/Glucose-Level-Prediction).

If you have questions and want to connect, you can message me on LinkedIn or Twitter. Also, follow me on Twitter @pacejohn and LinkedIn https://www.linkedin.com/in/john-pace-phd-20b87070/

#artificialintelligence #ai #machinelearning #ml #timeseries #lstm #arima #sarima #nvidia #dgx #ibm #powersystems #ac922 #docker #conda #neuralnetworks #RNN #recurrentneuralnetwork #diabetes #pandas #numpy #tensorflow #keras

​Special thanks to Jason Brownlee and Jinyu Xie!
PictureJason Brownlee

Brownlee, Jason.  11 Classical Time Series Forecasting Methods in Python (Cheat Sheet).  
https://machinelearningmastery.com/time-series-forecasting-methods-in-python-cheat-sheet/ 

​Brownlee, Jason.  How to Develop LSTM Models for Time Series Forecasting.

https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting/

PictureJinyu Xie

Jinyu Xie. Simglucose v0.2.1 - https://github.com/jxx123/simglucose

0 Comments

    Archives

    December 2020
    November 2020
    September 2020
    August 2020
    June 2020
    May 2020
    April 2020
    March 2020
    February 2020
    January 2020
    December 2019
    November 2019
    October 2019
    September 2019
    August 2019
    May 2019
    April 2019
    March 2019
    April 2018
    March 2018
    January 2018
    November 2017

    Tweets by pacejohn
Proudly powered by Weebly
  • Home
  • About
  • Contact