top of page

Work Plan

Weekly Reports & Future Planning

This page presents the work plan for the weeks to come. At the end of each week, an article detailing the progress of the study is added to this page. The articles will be accessible below.

 

The study is currently finished. The dissertation is written and awaits jury evaluation. 

Completed Tasks

  • Seamlessly configure, launch and scale Hadoop YARN clusters using OpenStack and Sahara

  • Develop a simple Proof-of-Concept to burst jobs to a simulated local "public" cloud based on simple hardcoded values

  • Improve the Proof-of-Concept to consider Hadoop YARN container utilization and cluster capacity 

  • Automate the uploading of PCAP files to HDFS from a specified directory

  • Automate the capturing and transporting of PCAP files using TCPDump to the directory used for the upload to HDFS

  • Automate the storage of input and output values for the eventual use of a Machine Learning algorithm to predict completion times (Future work)

  • Implement a simple job completion time predictor based on a previously known job profile

  • Implement a simple prediction algorithm for the time it takes to upload to HDFS 

  • Update the Proof-of-Concept to consider the simultaneous execution of two jobs

  • Update prediction algorithm for the HDFS upload time to consider changes in the upload speed

  • Add a tolerance value for the upload period threshold

  • Automate retrieval of job logs and map task statistics

 

© 2015 by Pedro Rocha Gonçalves. Proudly created with Wix.com

  • Facebook Clean
  • Google+ Clean
  • LinkedIn Clean
bottom of page