> Markdown version of [/videos/162-data-science-on-software-data?t=2](https://www.wearedevelopers.com/videos/162-data-science-on-software-data?t=2). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Data Science on Software Data

Struggling to convince management to fix technical debt? Discover how to use Python and Jupyter to turn version control history into data-driven arguments for structural investments.

- **Speakers:** [Markus Harrer](https://www.wearedevelopers.com/@markus-harrer)
- **Event:** World Congress 2021
- **Published:** July 30, 2021
- **Duration:** 48:10
- **URL:** https://www.wearedevelopers.com/videos/162-data-science-on-software-data

## Summary

Bridging the communication gap between development and management regarding legacy system modernization requires moving beyond standard static code analysis. While off-the-shelf tools easily address generic code quality questions, unique architectural challenges demand project-specific methodologies rooted in empirical software engineering. By applying software analytics—treating software artifacts like version control history, runtime coverage, and module dependencies as raw datasets—developers can quantify abstract technical debt and securely motivate structural investments to business stakeholders.

Adopting a reproducible data science workflow with tools like Python, Pandas, and Jupyter Notebooks empowers developers to build automated, transparent code analyses. This "computational notebook approach" enables the documentation of context, assumptions, and step-by-step calculations in a format that teams can review and integrate directly into continuous delivery pipelines. Whether parsing coverage data to uncover abandoned code, mapping the progress of massive architecture migrations, or utilizing graph databases for root-cause dependency tracking, custom analyses surface systemic insights entirely invisible to standard linters.

Crucially, standard empirical models are rarely directly transferable between unique projects; teams must adapt the methodology rather than expecting plug-and-play metrics. When mining software data, teams should avoid non-actionable or compliance-risky person-specific metrics, focusing instead on structural boundaries and overall business value. Ultimately, exploring software data without a goal is wasted effort—an analysis is only complete if it defer to practical next steps and triggers tangible code refactoring.

**Keywords:** software analytics, legacy system modernization, empirical software engineering, python data science tooling, computational notebook approach, jupyter notebooks, pandas dataframe analysis, automated architectural review, version control data mining, runtime coverage analysis, reproducible data science workflows, project-specific code metrics, structural technical debt, software modularization methodologies, graph database code analysis, performance bottleneck identification, continuous delivery analytics

## Chapters

1. **Analyzing legacy systems using software data** (00:02) — Applying data analysis approaches to extract insights and improve decisions regarding legacy software architecture.
1. **Competing priorities between technical improvements and feature development** (01:59) — Competing business priorities cause necessary architectural updates to struggle for budget against highly visible feature requests and defect resolution.
1. **Transferability challenges in standard software analytics models** (04:16) — Early empirical software engineering research demonstrated the inability to transfer quality metrics across different unique IT projects.
1. **Choosing software analytics over standard static analysis tools** (09:35) — Moving beyond standard style checkers involves building specific analytical views that address unique project problems.
1. **Baseline developer skills for software data science** (14:21) — Standard software developers already possess the required quantitative, programming, and domain expertise to perform data science.
1. **Establishing reproducible data science with openness and automation** (15:52) — Integrating data tools into the continuous delivery pipeline achieves transparent and automated code analysis.
1. **Extracting insights from various types of software data** (17:55) — Mining static, runtime, chronological, and community development data exposes abandoned code and structural hotspots.
1. **Building a data analysis stack with Python and Jupyter** (23:46) — Utilizing Python, pandas, and computational notebooks documents and executes analytical assumptions transparently.
1. **Analyzing production code coverage data using pandas** (29:22) — Reading Jacoco execution traces into data frames visualizes unused application packages via grouped bar charts.
1. **Utilizing graph analytics and Neo4j for structural dependencies** (36:21) — Querying an interconnected code base modeled inside a graph database identifies code smells and architectural bounds.
1. **Core takeaways for adopting software data analytics workflows** (39:05) — Effective data analysis relies on custom tooling and verifiable, action-driven results rather than generating unapplied metrics.
1. **Resolving real-world performance bottlenecks through targeted data analysis** (40:20) — Custom dependency graphing isolated the exact Java object duplications that choked application scalability in production.

## Related Moments

- [Introduction to analytical data formats for software developers](https://www.wearedevelopers.com/videos/100075-parquet-delta-iceberg-ducklake-an-introduction-for-developers) (from "Parquet, Delta, Iceberg & Ducklake - An introduction for developers")
- [Identifying core challenges in modern software development](https://www.wearedevelopers.com/videos/1023-from-zero-to-hero-launch-manage-your-cloud-apps-with-free-openshift-red-hat-developer-hub) (from "From Zero to Hero: Launch & Manage Your Cloud Apps with Free OpenShift & Red Hat Developer Hub")
- [Transitioning away from dominant management with objective analytics](https://www.wearedevelopers.com/videos/1101-how-is-ai-changing-the-leadership-role) (from "How is AI changing the leadership role?")
- [Unifying core software principles for better team performance](https://www.wearedevelopers.com/videos/721-the-quest-for-better-software) (from "The QUEST for Better Software")
- [Managing legacy infrastructure and distributed software systems](https://www.wearedevelopers.com/videos/100054-inside-mercedes-benz-140-years-of-heritage-meet-ai) (from "Inside Mercedes-Benz: 140 Years of Heritage meet AI")
- [Historical evolution of software delivery and agile methodologies](https://www.wearedevelopers.com/videos/1855-the-intent-engineer-closing-the-gap-between-business-engineering-manuel-klein) (from "The Intent Engineer: Closing the Gap Between Business & Engineering - Manuel Klein")

## Related Articles

- [Now is the time for industrialized software development](https://www.wearedevelopers.com/magazine/601-now-is-the-time-for-industrialized-software-development)
- [Transforming Software Development: The Role of AI and Developer Tools](https://www.wearedevelopers.com/magazine/527-transforming-software-development-the-role-of-ai-and-developer-tools)
- [Is Software Development Making the Climate Crisis Worse?](https://www.wearedevelopers.com/magazine/551-is-software-development-making-the-climate-crisis-worse)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)

## Related Jobs

- [Senior Software Engineer, Data](https://www.wearedevelopers.com/jobs/48273-senior-software-engineer-data) at **Sportradar Media Services GmbH**
- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Senior Data Engineer](https://www.wearedevelopers.com/jobs/ext/1589390-senior-data-engineer) at **Douglas GmbH**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Principal Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1465908-principal-software-engineer-database-infrastructure) at **GitHub**
- [Staff Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1470125-staff-software-engineer-database-infrastructure) at **GitHub**