EUVIMEDEUVIMED entdecken
Crossref

From Raw Trades to Audit-Ready Insights Designing Regulator-Grade Market Surveillance Pipelines

Janardhan Reddy Kasireddy

International Journal of Engineering & Extended Technologies Research · 2022

DOI: 10.15662/ijeetr.2022.0402003 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Engineering & Extended Technologies Research, 2022. Verfasst von Janardhan Reddy Kasireddy. Die Arbeit wurde bisher 1-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper proposes a holistic way of how to design end-to-end market surveillance data pipelines that convert raw trade and quote data into regulator grade analytics and at the same time achieve auditability, traceability and satisfaction of governance rules. We used the NYSE Trades and Quotes (TAQ) dataset available on Kaggle which consists of high-frequency trades and quotes for various equities to run a multi-step data engineering pipeline that includes ingestion, validation, transformation, enrichment and storage. The raw data is initially loaded in a distributed object store, and checked with regards to the schema consistency, time stamp synchronization, and completeness. At each stage, immutable logs are created, which help track the lineages and make audits ready. Apache Spark is used to perform data transformations such as aggregation, normalization, and anomaly detection on a scalable cluster architecture through which multi-terabyte datasets can be processed in a parallel manner. As can be seen in the evaluation, our pipeline can ingest live trade streams with a latency of less than a second, and still show a complete trace of raw trading data to analytical results. Lineage and immutable logging allow the auditors to recreate any derived metric, which meets the requirements of regulators to be accountable and transparent. This paper provides a repeatable system to finance institutions who want to operationalize market surveillance analytics at scale by systematically integrating distributed processing, strong data governance and end to end lineage.

Zitieren

APA

Kasireddy, J. R. (2022). From Raw Trades to Audit-Ready Insights Designing Regulator-Grade Market Surveillance Pipelines. International Journal of Engineering & Extended Technologies Research. https://doi.org/10.15662/ijeetr.2022.0402003

Vancouver

Kasireddy JR. From Raw Trades to Audit-Ready Insights Designing Regulator-Grade Market Surveillance Pipelines. International Journal of Engineering & Extended Technologies Research. 2022. doi:10.15662/ijeetr.2022.0402003

BibTeX
@article{Kasireddy2022,
  author = {Kasireddy, Janardhan Reddy},
  title = {From Raw Trades to Audit-Ready Insights Designing Regulator-Grade Market Surveillance Pipelines},
  journal = {International Journal of Engineering & Extended Technologies Research},
  year = {2022},
  doi = {10.15662/ijeetr.2022.0402003}
}
Kostenlos starten →