EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Research on Construction of Offline Data Warehouse for Ship Shore Power based on DolphinScheduler and Hive

Zewen Zhang, Xin Zhang, Taizhi Lv

Frontiers in Science and Engineering · 2024

DOI: 10.54691/23cqqy15 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Science and Engineering, 2024. Verfasst von Zewen Zhang und 2 weiteren Autor:innen. Die Arbeit nennt 7 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In today's networked, intelligent, and data-driven era, the shore power industry is facing the challenge of rapidly growing data. This paper presents the construction of a professional offline data warehouse system for shore power based on DolphinScheduler and Hive. Firstly, MySQL is adopted as the backend database, combined with Sqoop to synchronize business data to HDFS, ensuring data reliability and integrity. Secondly, the Flume-Kafka-Flume architecture is utilized to achieve real-time collection and caching of user behavior data, providing data support for subsequent analysis. Thirdly, HQL statements are written in Hive to clean, merge, and analyze shore power data, calculating key indicators such as electricity consumption and usage trends. Fourthly, data visualization is achieved through the integration of Superset, displaying data analysis results via a web interface. Fifthly, DolphinScheduler is employed for timed scheduling, ensuring dependency control among various tasks and the smooth operation of the project. This system fully leverages the replication mechanism of HDFS to enhance reliability, dynamically adds nodes to achieve system scalability, and fully utilizes the fault tolerance of the Yarn scheduler. It saves time and computational costs for the shore power industry, realizing higher value and benefits.

Zitieren

APA

Zhang, Z., Zhang, X., & Lv, T. (2024). Research on Construction of Offline Data Warehouse for Ship Shore Power based on DolphinScheduler and Hive. Frontiers in Science and Engineering. https://doi.org/10.54691/23cqqy15

Vancouver

Zhang Z, Zhang X, Lv T. Research on Construction of Offline Data Warehouse for Ship Shore Power based on DolphinScheduler and Hive. Frontiers in Science and Engineering. 2024. doi:10.54691/23cqqy15

BibTeX
@article{Zhang2024,
  author = {Zhang, Zewen and Zhang, Xin and Lv, Taizhi},
  title = {Research on Construction of Offline Data Warehouse for Ship Shore Power based on DolphinScheduler and Hive},
  journal = {Frontiers in Science and Engineering},
  year = {2024},
  doi = {10.54691/23cqqy15}
}
Kostenlos starten →