Our group develops deep-learning tools that turn routine clinical CT into quantitative inputs for fracture-risk assessment. Working with radiation oncologists, radiologists, and computer scientists at Dana-Farber Cancer Institute, Brigham and Women's Hospital, and Isomics, Inc., we are building and optimizing the components to allow an integrated pipeline that will provide automated segmentation of the cancer spine, objectively detect and label metastatic vertebrae containing bone metastases, classify the type of bone lesion and track its growth over time to quantify tumor aggressiveness and response to treatment (radiotherapy, chemotherapy, and immunotherapy) and baseline and evolving risk of vertebral fracture and neurological compromise and as a driver of our finite element biomechanical studies. Our work has similarly focused on the development of automated segmentation of the thoracic and lumbar musculature 3D anatomy and structure to support our research using musculoskeletal spine models. These developments drive our goal of individualized prediction of vertebral fracture risk in patients with cancer.
Deep-Learning models
Development of cancer spine specific registration methods
Tracking lesion growth over time
Monitoring how a lesion evolves after treatment requires spatially aligning the longitudinal CT scans acquired months apart, often on different scanners and with different fields of view. Standard intensity-based registration struggles in metastatic vertebrae, whose shape and density can change substantially between scans. We developed a two-step approach [2]: the deep-learning model first segments and labels each vertebra, and the resulting 3D surfaces are aligned vertebra by vertebra using a rigid, Gaussian mixture model surface registration. Because the alignment relies on shape rather than intensity, it remains robust as lesions grow.
[A] | 
[B] |
|---|
Figure 1 Tracking Lesion changes at baseline, 3 and 6 months, fused with the longitudinal CT [A] with a comparison of 12 Months with baseline showing lesion growth visualization enabled by the registeration [2]
Automated segmentation and longitudinal registration of metastatic spines
Scaling this work up to a full benchmark of registration methods for cancer spines [5] in collaboration with Technical University of Munich, we evaluate eleven rigid and deformable methods, including classical approaches and deep-learning models (ANTs, deepali, uniGradICON, VoxelMorph), on a sample of 200 cancer patients' spines at baseline and at 3, 6, and 12 months, yielding 2,986 vertebra-level registration pairs [5]. Registering each vertebra separately, rather than the whole spine at once, substantially improved alignment, and deformable methods outperformed rigid ones. The optimization-based deepali method performed best, with a mean Dice score of 0.953 and a mean target registration error of 1.1 mm. It aligned even collapsed or cement-augmented vertebrae well. The benchmark also showed that the disease itself erodes registration accuracy. In vertebrae with lesions, accuracy fell steadily from 3 to 12 months, much more than in uninvolved vertebrae, as lesions grew and fractures, radiotherapy, and cement altered the bone.
[A] | 
[B] |
|---|
Figure 2. A severely deformed vertebra before and after deformable registration [A]. Our work further highlights the challenge posed by the effect of the metastatic lesions on vertebral geometry and appearance for longitudinal evaluation of pathologic vertebrae [B] (Rehtanz 2025, [5]).
We are working to package the best-performing methods as an open-source 3D Slicer extension to allow clinicians and researchers to automatically segment and register full spine or selected vertebrae in cancer patients with metastatic spine disease for further evaluation and the creation of computational models without writing code.
Classifying bone lesion type
Whether a metastasis is osteolytic, osteoblastic, or mixed strongly influences fracture risk and treatment choice, yet this classification is subjective and agreement among specialists is only poor to moderate. Training a classifier is also hampered by the lack of voxel-level lesion outlines; clinical annotations are typically made per vertebra.
Vertebral level classification
We introduced a 3D convolutional neural network strategy that learns from these vertebra-level labels [4]. Every voxel in an annotated vertebra is assigned that vertebra's lesion class, an nnU-Net model is trained on the resulting coarse label maps, and its voxel-wise predictions are aggregated into a single class for each vertebra. Because it analyzes the whole vertebral volume, the model can recognize mixed lesions whose lytic and blastic components may not appear on any single slice.
In 2,125 vertebrae from 151 radiotherapy patients, the model reached 84.7% accuracy on an independent test set, compared with 72.1% for a conventional DenseNet121 classifier. Accuracy reached 91.7% for mixed lesions and 96.9% for vertebrae without lesions. Its agreement with a blinded musculoskeletal radiologist (Cohen's kappa 0.76) was comparable to agreement between radiologists, and its overall accuracy approached the radiologist's (87.1%).
Figure 3: Example CT and classifier prediction maps for no lesion, osteolytic, osteoblastic, and mixed vertebrae [4]. Sections I-IV show the CT data with sections V-VII the model labeling per lesion class.
Small, diffuse osteolytic lesions remained difficult for both the model and the radiologist to distinguish from age- and treatment-related bone loss, and this is a focus of ongoing work.
3D Segmentation of the trunk muscles
A vertebra fractures when the load placed on it exceeds its strength, and much of that load comes from the trunk muscles. Patient-specific assessment of task- and spinal-level loading requires personalized musculoskeletal models able to simulate the task evaluated. For this purpose, the models need the geometry of each vertebra and muscle along the thoracic and lumbar spine. However, no 3D training data exist for these muscles. Manually segmenting a single T4 to L4 spine would take an estimated 12 to 18 hours; the process is prone to error and is a time-prohibitive task for patient studies with single or longitudinal data.
This project utilized A 2D nnU-Net trained on single strong annotations, created for eight flexor, extensor, and stabilizing muscles per side, at a single axial slice per vertebral level from T4 to L4, about 2% of the slices in each scan. Trained as a sparse training set, the model provides complete volume segmentation of the entire trunk musculature in about one minute per patient.
Figure 4.per-level 2D segmentation [B} of the patients thoracic musculature on CT data [A] and the resulting 3D muscle volume [C}. Combine with the muscle CT density allows direct visualization of the muscle 3D structure [D]
Current directions
- Integrating segmentation, lesion classification, and longitudinal registration into a single pipeline linked to fracture-risk prediction.
- Combining imaging with clinical data (age, primary cancer, treatment history) and radiomics to improve classification and change in tissue parameters of metastatic bone lesions.
- Building a larger longitudinal CT dataset to quantify how metastatic bone lesions shape, structure and growth relate to fracture risk both pre- and post treatment radiotherapy.
- Expert, multi-reader lesion segmentation to train reliable automated lesion models, and extending CT-to-CT registration to combined CT and MRI.
- Fully 3D muscle segmentation for spines with scoliosis and other large deformities.
- Assessment of patient frailty and its association with falls and fall related spinal fractures.
- Multi-institutional validation across CT scanners and acquisition protocols.
Publications
- Haouchine N, Hackney DB, Pieper SD, Wells WM III, Sanhinova M, Balboni TA, Spektor A, Huynh MA, Kozono DE, Doyle P, Alkalay RN. An open annotated dataset and baseline machine learning model for segmentation of vertebrae with metastatic bone lesions from CT. medRxiv 2024. doi:10.1101/2024.10.14.24314447
- Sanhinova M, Haouchine N, Pieper SD, Wells WM III, Balboni TA, Spektor A, Huynh MA, Guenette JP, Czajkowski B, Caplan S, Doyle P, Kang H, Hackney DB, Alkalay RN. Registration of longitudinal spine CTs for monitoring lesion growth. Proc. SPIE Medical Imaging 2024: Image Processing. 2024;12926:658–661. arXiv:2402.09341
- Hong V, Pieper S, James J, Anderson DE, Pinter C, Chang YS, Bulent A, Kozono D, Doyle P, Caplan S, Kang H, Balboni T, Spektor A, Keko M, Kikinis R, Hackney DB, Alkalay RN. Automated segmentation of trunk musculature with a deep CNN trained from sparse annotations in radiation therapy patients with metastatic spine disease: an observational study. Front Bioeng Biotechnol. 2026;13:1707724. doi:10.3389/fbioe.2025.1707724
- Hong V, Aslan B, Haouchine N, Pieper S, Wells W, Keko M, Kozono D, Doyle PF, Balboni T, Spektor A, Huynh MA, Hackney DB, Alkalay RN. Deep learning-based classification of bone lesions on CT scans of metastatic spine disease patients: a 3D-convolutional neural network approach. Submitted to Journal of Spine: Imaging Cancer; preprint on medRxiv, 2026. [doi:10.64898/2026.08.13.26360407 .]
- Rehtanz NJ. Tracking tumor-induced structural changes in the spine via longitudinal CT registration. Master's thesis, Information Systems, Technical University of Munich; 2025. Supervisor: Prof. D. Rückert; advisors: R. Graf, M.Sc., and R.N. Alkalay, PhD.
