DATA ENGINEERING
Data preparation
Quality checks and reproducible transformation of the source dataset.
PythonpandasNumPyExcel / CSV
Project year2024
MEDCYB
MEDICAL AI · DATA SCIENCE · RESEARCH
MEDCYB
Machine-learning analysis of 15,000+ clinical observations and about 85 source parameters to identify patterns associated with gestational diabetes risk.
Clinical research · MEDCYB
The question behind the dataset
Gestational diabetes risk depends on many factors. Together with the obstetrics and gynaecology department of St. Petersburg Pediatric University, I set up a research workflow to examine clinical observations and test candidate predictors.
Dataset
15 000+
clinical observations
Input
~85
source parameters
Quality criterion
The prepared dataset and the origin of each feature must remain traceable.
Research protocol
From clinical records to auditable predictors
The value is not a single model score. It is a traceable path from the source table through quality checks and feature preparation to validation and clinical interpretation.
01
Map the source dataset
Check the clinical table structure, field types and availability of the target variable.
Data structure02
Check data quality
Find missing values, duplicates, outliers and inconsistent category values.
Quality issues03
Prepare the dataset
Define reproducible cleaning and transformation rules while preserving feature provenance.
Prepared data04
Build the features
Explore distributions and relationships, prepare numerical and categorical features, and check for leakage.
Feature matrix05
Compare approaches
Compare classification approaches on held-out data and examine missed meaningful cases separately.
Validation result06
Interpret the findings
Identify patterns and predictors that can be discussed substantively with medical specialists.
Auditable predictorsTools and methods
DATA · SCIENCE · MLDATA SCIENCE
Exploratory analysis
Distributions, anomalies and relationships between clinical parameters.
JupyterMatplotlibEDAData quality
MACHINE LEARNING
Models and validation
Feature selection, classification and evaluation on held-out data.
scikit-learnFeature selectionClassificationValidation
Result and scope
A basis for clinical discussion, not a diagnosis
The analysis surfaced patterns and predictors associated with gestational diabetes risk. The prepared feature matrix and the reasoning behind the classification can be reviewed with medical specialists; the work remains in a research setting and does not replace clinical judgement.
Have a complex data or AI research task?
Let’s review the source data, constraints and risks, then define a useful first stage that can actually be measured.