A framework for real time IOT stream processing with a large number of sensor specific data pipelines

Inspired by the current growth of IoT, this thesis investigates opportunities of gaining insights on data generated by a large number of weather sensors. As the increasing amount of data poses new challenges on processing this data in real-time, we compare classical big batch processing to continuous stream processing. After investigating different tools to process data in real-time, a experimental prototype is implemented with the goal of providing reusable data pipelines to transform streamed weather data. Overall, this prototype proves to be a good starting point as a classification and offering the ability to execute pre-defined as well as user-defined transformations in parallel pipelines. Future research is warranted to improve performance and validation of these configured data pipelines

Project information



Thesis for degree:



Phillip Kemper