Model

How the model measures a hip

The model is a small neural network. It was trained on 370 X-rays that the surgeon had already marked by hand. To test it fairly we split the cases into five groups, trained on four and tested on the fifth, so it was always judged on cases it had never seen.

It finds two points: the tip of the screw and the apex of the femoral head. It does not guess how wide the screw is. We measure that straight from the image, by finding where the bright metal ends.

  1. Phone photo
  2. 1Find the screw
  3. 2Zoom in
  4. 3Find tip and apex
  5. 4Measure the width
  6. 5Convert to mm

The measurement

What the surgeon marks

On each X-ray the surgeon marks the tip of the screw, the apex of the femoral head, and the two edges of the screw. The apex is where the line of the screw, carried on, meets the edge of the head.

apextipwidthAPapextipwidthLAT

mm = tip to apex in px × real screw width ÷ screw width in px

TAD = AP mm + LAT mm

Above 25 mm the screw is more likely to cut out of the head, so 25 mm is the number that matters.

How it works

From a phone photo to a number

STEP 1

Find the screw

60% of the long side

The photos show the whole monitor, so the hip can be anywhere. A first model looks at the whole photo and makes a rough guess of the four points. We keep only their centre and cut a square around it.

STEP 2

Zoom in until it settles

123

A second, more precise model looks inside the square and finds the points again. We cut a tighter square around its answer and repeat. It stops when the square moves less than 3 pixels, or after 6 rounds.

STEP 3

Find the tip and apex

256 × 256ResNet-183 × uptipapexwidth 1width 2

The final crop is resized to 256 × 256 pixels. The network returns four small maps, one per point, that light up where it thinks the point is. We take the brightest spot. On the server, two trained copies of the model average their maps first.

STEP 4

Measure the width

10 lines across the screwhalf waydarkbrightbrightness

The model's two width points only tell us where along the screw to look. We read the brightness across the screw on 10 lines and find where it crosses half way between bone and metal. The middle of the 10 results is the width in pixels.

STEP 5

Turn pixels into millimetres

The real width of the screw is entered with each case. Comparing it with the width in pixels gives the scale for that X-ray.

Example with typical numbers from our photos

AP19 px × 10.3 mm ÷ 21 px = 9.32 mm

LAT18 px × 10.3 mm ÷ 20 px = 9.27 mm

TAD9.32 + 9.27 = 18.59 mm

Training

How it was trained

We started from a network already trained on a large set of everyday photos, then taught it with the surgeon's marks: 185 cases, two views each, four points per view.

A changed copy every time

Each time it sees an X-ray during training, it gets a slightly changed copy. This teaches it what a screw looks like, not what one particular photo looks like.

original
turned ±25°
mirrored
inverted
blurred
brighter, grainy

A fair test

The cases were split into five groups of 37. Each round trains on four groups and tests on the fifth, so every case is scored by a model that never saw it. The two views of a case always stay in the same group.

Results

How close it gets

These results are from all 184 cases with both views. Each was scored by a model that had never seen it, starting from the full photo with no help.

4.31

mm average error

2.95

mm typical error

1.9

px typical tip error

12%

over 25 mm caught

Surgeon under 25 mmSurgeon over 25 mm (17)
001010202030304040agreemissed 15caught 2false alarm 11surgeon, mmmodel, mm
One case is off the top of the chart: surgeon 30.5 mm, model 72.8 mm. It counts as caught, but the reading is wrong.
Show as a table
#Surgeon mmModel mmDifference
16.307.56+1.26
28.3814.43+6.05
39.2811.73+2.45
49.5911.01+1.42
59.6422.39+12.75
610.6822.36+11.68
711.0310.94-0.09
811.4617.42+5.96
911.4714.44+2.97
1011.5914.38+2.79
1111.6016.77+5.17
1211.6116.56+4.95
1311.9716.35+4.38
1412.0812.91+0.83
1512.2514.50+2.25
1612.3112.96+0.65
1712.3511.56-0.79
1812.3914.45+2.06
1912.4211.49-0.93
2012.4616.03+3.57
2112.6113.09+0.48
2212.9112.14-0.77
2312.9315.36+2.43
2412.9316.81+3.88
2513.0518.34+5.29
2613.1917.17+3.98
2713.3614.75+1.39
2813.3714.87+1.50
2913.4217.39+3.97
3013.4710.43-3.04
3113.4717.06+3.59
3213.7615.38+1.62
3313.7615.87+2.11
3413.8912.71-1.18
3513.9512.46-1.49
3613.9513.24-0.71
3714.0014.64+0.64
3814.1911.85-2.34
3914.1915.35+1.16
4014.2114.87+0.66
4114.2314.29+0.06
4214.2516.44+2.19
4314.2815.18+0.90
4414.2914.26-0.03
4514.3616.63+2.27
4614.3713.32-1.05
4714.4116.44+2.03
4814.5415.04+0.50
4914.6713.46-1.21
5014.7313.91-0.82
5114.7518.27+3.52
5214.7914.76-0.03
5314.9016.00+1.10
5415.0028.28+13.28
5515.0411.02-4.02
5615.1112.17-2.94
5715.2114.16-1.05
5815.2713.52-1.75
5915.2714.33-0.94
6015.2914.57-0.72
6115.3215.92+0.60
6215.4417.03+1.59
6315.4913.45-2.04
6415.6716.50+0.83
6515.7513.09-2.66
6615.7717.54+1.77
6715.8416.93+1.09
6815.8616.19+0.33
6915.8916.26+0.37
7015.8916.92+1.03
7115.9414.33-1.61
7215.9716.77+0.80
7316.1413.89-2.25
7416.1415.30-0.84
7516.2416.15-0.09
7616.2814.04-2.24
7716.3714.66-1.71
7816.3729.12+12.75
7916.3817.33+0.95
8016.5019.62+3.12
8116.6214.59-2.03
8216.6317.85+1.22
8316.6415.60-1.04
8416.7017.09+0.39
8516.7219.68+2.96
8616.7410.75-5.99
8716.9417.28+0.34
8816.9916.01-0.98
8917.1425.07+7.93
9017.3614.26-3.10
9117.3711.82-5.55
9217.4111.30-6.11
9317.4413.46-3.98
9417.4712.13-5.34
9517.5817.53-0.05
9617.6315.14-2.49
9717.7325.37+7.64
9817.7914.38-3.41
9917.9516.18-1.77
10017.9516.26-1.69
10117.9921.05+3.06
10218.0517.05-1.00
10318.0517.11-0.94
10418.2614.28-3.98
10518.3115.03-3.28
10618.4215.52-2.90
10718.5014.96-3.54
10818.5430.42+11.88
10918.6217.57-1.05
11018.7517.37-1.38
11118.8531.16+12.31
11218.8618.79-0.07
11318.8914.32-4.57
11419.149.90-9.24
11519.2618.29-0.97
11619.2816.48-2.80
11719.2928.48+9.19
11819.319.85-9.46
11919.3214.09-5.23
12019.3728.70+9.33
12119.4315.56-3.87
12219.6417.98-1.66
12319.6917.67-2.02
12419.7011.73-7.97
12519.7012.44-7.26
12619.8218.53-1.29
12720.0818.62-1.46
12820.3117.80-2.51
12920.4218.09-2.33
13020.4914.14-6.35
13120.5220.23-0.29
13220.5814.48-6.10
13320.5917.65-2.94
13420.6316.57-4.06
13520.7415.05-5.69
13620.8715.98-4.89
13721.0415.23-5.81
13821.1316.09-5.04
13921.2115.86-5.35
14021.2416.76-4.48
14121.2517.01-4.24
14221.2834.37+13.09
14321.3918.90-2.49
14421.4229.92+8.50
14521.4716.17-5.30
14621.4817.78-3.70
14721.7116.93-4.78
14821.7317.88-3.85
14922.0217.52-4.50
15022.0613.41-8.65
15122.1424.24+2.10
15222.1620.58-1.58
15322.3813.17-9.21
15422.3818.53-3.85
15522.4714.62-7.85
15622.4917.96-4.53
15722.5417.62-4.92
15822.5518.66-3.89
15922.9715.02-7.95
16023.1122.77-0.34
16123.1219.33-3.79
16223.3137.28+13.97
16323.3717.03-6.34
16423.6613.55-10.11
16523.7717.80-5.97
16624.2613.11-11.15
16724.7016.09-8.61
16825.2416.31-8.93
16925.6519.33-6.32
17025.7614.62-11.14
17126.1026.51+0.41
17226.3320.66-5.67
17326.5324.65-1.88
17426.6916.54-10.15
17527.0218.46-8.56
17627.0519.24-7.81
17727.2217.59-9.63
17827.6816.40-11.28
17928.0120.82-7.19
18030.4772.79+42.32
18130.9419.30-11.64
18231.9114.17-17.74
18332.2518.32-13.93
18433.1516.47-16.68

Reading the chart

Each dot is one case. On the line, the model and the surgeon agree.

How often it is close

67 of 184 cases are within 2 mm, 94 within 3 mm and 127 within 5 mm.

Where it goes wrong

It pulls answers toward the middle. Below 15 mm it reads about 2 mm high. Above 25 mm it reads about 6 mm low, which is why it caught only 2 of the 17 cases the surgeon measured above 25 mm.

The points themselves

The tip is typically 1.9 px from the surgeon’s mark and the apex 4.1 px. The measured width is within 10% of the surgeon’s on 305 of 368 X-rays.

What we tried

How we got here

average error, mm
01

We asked an AI to read the distance

Its answers stayed close to the 25 mm written in the question.

9.38

20 cases

02

We asked it for pixel positions instead

Better, but not repeatable. The same 10 cases scored 3.73 mm, then 5.60 mm on a second run.

3.73

10 cases

03

We trained our own model on the surgeon's marks

It found the tip and apex, but not the width of the screw.

109

159 cases

04

We measured the width from the image

Tested on crops drawn around the surgeon's marks, which a new case does not have.

3.40

185 cases

05

We let it find the screw on its own

The whole photo goes in with no help. This is what runs today.

4.31

184 cases

Rows 01 and 02 were tested on few cases because each run cost money. Row 04 scores better than row 05 because it was helped by the surgeon's marks.

Limits

What it cannot do yet

Make the 25 mm call

It caught 2 of the 17 cases the surgeon measured above 25 mm. It should not be used to decide on its own.

Tell us how much surgeons disagree

One surgeon marked every case, so there is no second reading to compare against yet.

Get past the photo

In a phone photo of the monitor the screw is only about 20 pixels wide. A photo zoomed in on the hip would help every method, including a person.

Why a few pixels matter

19 pxtip ± 2 pxapex ± 2 px

In a typical photo the tip and apex are about 19 pixels apart. If every point were off by 2 pixels, even a perfect reader would be about 1.8 mm out on average.

Technical details

The system

Network
ResNet-18 with 3 upsampling layers (the "SimpleBaseline" pose model)
Starting point
Weights pre-trained on ImageNet
Input and output
256 × 256 greyscale crop in, four 64 × 64 heat maps out
Training data
370 X-rays from 185 cases, 4 points each, marked by one surgeon
Training
90 epochs, AdamW, learning rate 3e-4 on a one-cycle schedule, batch 16, mean squared error on the heat maps
Locator model
Same network on the whole photo, 70 epochs, 3 groups
Changed copies
Turn ±25°, mirror 50%, invert 35%, blur 30%, contrast 0.75 to 1.3×, brightness ±12%, grain; crop size 0.75 to 1.55×, shift ±16%
Testing
5-fold cross-validation, split by case, 37 cases per group
Width
Full width at half maximum, middle value of 10 profiles over ±14 px
Runs on
ONNX Runtime on the CPU of our own server. No image goes to an outside AI service, and a case costs nothing to run.
Live version
Compressed (int8) copies of 1 locator and 2 keypoint models, about 250 MB of memory
Training time
About 75 minutes on an Apple M1 Max

The scores above come from full-size models, each tested only on cases it did not train on. The server runs the smaller compressed copies to fit in memory. Those were checked on 6 cases, not on all 184.

Author

Julien Hosri

Product designer and engineer