STEP 1
Find the screw
The photos show the whole monitor, so the hip can be anywhere. A first model looks at the whole photo and makes a rough guess of the four points. We keep only their centre and cut a square around it.
Model
The model is a small neural network. It was trained on 370 X-rays that the surgeon had already marked by hand. To test it fairly we split the cases into five groups, trained on four and tested on the fifth, so it was always judged on cases it had never seen.
It finds two points: the tip of the screw and the apex of the femoral head. It does not guess how wide the screw is. We measure that straight from the image, by finding where the bright metal ends.
The measurement
On each X-ray the surgeon marks the tip of the screw, the apex of the femoral head, and the two edges of the screw. The apex is where the line of the screw, carried on, meets the edge of the head.
mm = tip to apex in px × real screw width ÷ screw width in px
TAD = AP mm + LAT mm
Above 25 mm the screw is more likely to cut out of the head, so 25 mm is the number that matters.
How it works
STEP 1
The photos show the whole monitor, so the hip can be anywhere. A first model looks at the whole photo and makes a rough guess of the four points. We keep only their centre and cut a square around it.
STEP 2
A second, more precise model looks inside the square and finds the points again. We cut a tighter square around its answer and repeat. It stops when the square moves less than 3 pixels, or after 6 rounds.
STEP 3
The final crop is resized to 256 × 256 pixels. The network returns four small maps, one per point, that light up where it thinks the point is. We take the brightest spot. On the server, two trained copies of the model average their maps first.
STEP 4
The model's two width points only tell us where along the screw to look. We read the brightness across the screw on 10 lines and find where it crosses half way between bone and metal. The middle of the 10 results is the width in pixels.
STEP 5
The real width of the screw is entered with each case. Comparing it with the width in pixels gives the scale for that X-ray.
Example with typical numbers from our photos
AP19 px × 10.3 mm ÷ 21 px = 9.32 mm
LAT18 px × 10.3 mm ÷ 20 px = 9.27 mm
TAD9.32 + 9.27 = 18.59 mm
Training
We started from a network already trained on a large set of everyday photos, then taught it with the surgeon's marks: 185 cases, two views each, four points per view.
Each time it sees an X-ray during training, it gets a slightly changed copy. This teaches it what a screw looks like, not what one particular photo looks like.
The cases were split into five groups of 37. Each round trains on four groups and tests on the fifth, so every case is scored by a model that never saw it. The two views of a case always stay in the same group.
Results
These results are from all 184 cases with both views. Each was scored by a model that had never seen it, starting from the full photo with no help.
4.31
mm average error
2.95
mm typical error
1.9
px typical tip error
12%
over 25 mm caught
| # | Surgeon mm | Model mm | Difference |
|---|---|---|---|
| 1 | 6.30 | 7.56 | +1.26 |
| 2 | 8.38 | 14.43 | +6.05 |
| 3 | 9.28 | 11.73 | +2.45 |
| 4 | 9.59 | 11.01 | +1.42 |
| 5 | 9.64 | 22.39 | +12.75 |
| 6 | 10.68 | 22.36 | +11.68 |
| 7 | 11.03 | 10.94 | -0.09 |
| 8 | 11.46 | 17.42 | +5.96 |
| 9 | 11.47 | 14.44 | +2.97 |
| 10 | 11.59 | 14.38 | +2.79 |
| 11 | 11.60 | 16.77 | +5.17 |
| 12 | 11.61 | 16.56 | +4.95 |
| 13 | 11.97 | 16.35 | +4.38 |
| 14 | 12.08 | 12.91 | +0.83 |
| 15 | 12.25 | 14.50 | +2.25 |
| 16 | 12.31 | 12.96 | +0.65 |
| 17 | 12.35 | 11.56 | -0.79 |
| 18 | 12.39 | 14.45 | +2.06 |
| 19 | 12.42 | 11.49 | -0.93 |
| 20 | 12.46 | 16.03 | +3.57 |
| 21 | 12.61 | 13.09 | +0.48 |
| 22 | 12.91 | 12.14 | -0.77 |
| 23 | 12.93 | 15.36 | +2.43 |
| 24 | 12.93 | 16.81 | +3.88 |
| 25 | 13.05 | 18.34 | +5.29 |
| 26 | 13.19 | 17.17 | +3.98 |
| 27 | 13.36 | 14.75 | +1.39 |
| 28 | 13.37 | 14.87 | +1.50 |
| 29 | 13.42 | 17.39 | +3.97 |
| 30 | 13.47 | 10.43 | -3.04 |
| 31 | 13.47 | 17.06 | +3.59 |
| 32 | 13.76 | 15.38 | +1.62 |
| 33 | 13.76 | 15.87 | +2.11 |
| 34 | 13.89 | 12.71 | -1.18 |
| 35 | 13.95 | 12.46 | -1.49 |
| 36 | 13.95 | 13.24 | -0.71 |
| 37 | 14.00 | 14.64 | +0.64 |
| 38 | 14.19 | 11.85 | -2.34 |
| 39 | 14.19 | 15.35 | +1.16 |
| 40 | 14.21 | 14.87 | +0.66 |
| 41 | 14.23 | 14.29 | +0.06 |
| 42 | 14.25 | 16.44 | +2.19 |
| 43 | 14.28 | 15.18 | +0.90 |
| 44 | 14.29 | 14.26 | -0.03 |
| 45 | 14.36 | 16.63 | +2.27 |
| 46 | 14.37 | 13.32 | -1.05 |
| 47 | 14.41 | 16.44 | +2.03 |
| 48 | 14.54 | 15.04 | +0.50 |
| 49 | 14.67 | 13.46 | -1.21 |
| 50 | 14.73 | 13.91 | -0.82 |
| 51 | 14.75 | 18.27 | +3.52 |
| 52 | 14.79 | 14.76 | -0.03 |
| 53 | 14.90 | 16.00 | +1.10 |
| 54 | 15.00 | 28.28 | +13.28 |
| 55 | 15.04 | 11.02 | -4.02 |
| 56 | 15.11 | 12.17 | -2.94 |
| 57 | 15.21 | 14.16 | -1.05 |
| 58 | 15.27 | 13.52 | -1.75 |
| 59 | 15.27 | 14.33 | -0.94 |
| 60 | 15.29 | 14.57 | -0.72 |
| 61 | 15.32 | 15.92 | +0.60 |
| 62 | 15.44 | 17.03 | +1.59 |
| 63 | 15.49 | 13.45 | -2.04 |
| 64 | 15.67 | 16.50 | +0.83 |
| 65 | 15.75 | 13.09 | -2.66 |
| 66 | 15.77 | 17.54 | +1.77 |
| 67 | 15.84 | 16.93 | +1.09 |
| 68 | 15.86 | 16.19 | +0.33 |
| 69 | 15.89 | 16.26 | +0.37 |
| 70 | 15.89 | 16.92 | +1.03 |
| 71 | 15.94 | 14.33 | -1.61 |
| 72 | 15.97 | 16.77 | +0.80 |
| 73 | 16.14 | 13.89 | -2.25 |
| 74 | 16.14 | 15.30 | -0.84 |
| 75 | 16.24 | 16.15 | -0.09 |
| 76 | 16.28 | 14.04 | -2.24 |
| 77 | 16.37 | 14.66 | -1.71 |
| 78 | 16.37 | 29.12 | +12.75 |
| 79 | 16.38 | 17.33 | +0.95 |
| 80 | 16.50 | 19.62 | +3.12 |
| 81 | 16.62 | 14.59 | -2.03 |
| 82 | 16.63 | 17.85 | +1.22 |
| 83 | 16.64 | 15.60 | -1.04 |
| 84 | 16.70 | 17.09 | +0.39 |
| 85 | 16.72 | 19.68 | +2.96 |
| 86 | 16.74 | 10.75 | -5.99 |
| 87 | 16.94 | 17.28 | +0.34 |
| 88 | 16.99 | 16.01 | -0.98 |
| 89 | 17.14 | 25.07 | +7.93 |
| 90 | 17.36 | 14.26 | -3.10 |
| 91 | 17.37 | 11.82 | -5.55 |
| 92 | 17.41 | 11.30 | -6.11 |
| 93 | 17.44 | 13.46 | -3.98 |
| 94 | 17.47 | 12.13 | -5.34 |
| 95 | 17.58 | 17.53 | -0.05 |
| 96 | 17.63 | 15.14 | -2.49 |
| 97 | 17.73 | 25.37 | +7.64 |
| 98 | 17.79 | 14.38 | -3.41 |
| 99 | 17.95 | 16.18 | -1.77 |
| 100 | 17.95 | 16.26 | -1.69 |
| 101 | 17.99 | 21.05 | +3.06 |
| 102 | 18.05 | 17.05 | -1.00 |
| 103 | 18.05 | 17.11 | -0.94 |
| 104 | 18.26 | 14.28 | -3.98 |
| 105 | 18.31 | 15.03 | -3.28 |
| 106 | 18.42 | 15.52 | -2.90 |
| 107 | 18.50 | 14.96 | -3.54 |
| 108 | 18.54 | 30.42 | +11.88 |
| 109 | 18.62 | 17.57 | -1.05 |
| 110 | 18.75 | 17.37 | -1.38 |
| 111 | 18.85 | 31.16 | +12.31 |
| 112 | 18.86 | 18.79 | -0.07 |
| 113 | 18.89 | 14.32 | -4.57 |
| 114 | 19.14 | 9.90 | -9.24 |
| 115 | 19.26 | 18.29 | -0.97 |
| 116 | 19.28 | 16.48 | -2.80 |
| 117 | 19.29 | 28.48 | +9.19 |
| 118 | 19.31 | 9.85 | -9.46 |
| 119 | 19.32 | 14.09 | -5.23 |
| 120 | 19.37 | 28.70 | +9.33 |
| 121 | 19.43 | 15.56 | -3.87 |
| 122 | 19.64 | 17.98 | -1.66 |
| 123 | 19.69 | 17.67 | -2.02 |
| 124 | 19.70 | 11.73 | -7.97 |
| 125 | 19.70 | 12.44 | -7.26 |
| 126 | 19.82 | 18.53 | -1.29 |
| 127 | 20.08 | 18.62 | -1.46 |
| 128 | 20.31 | 17.80 | -2.51 |
| 129 | 20.42 | 18.09 | -2.33 |
| 130 | 20.49 | 14.14 | -6.35 |
| 131 | 20.52 | 20.23 | -0.29 |
| 132 | 20.58 | 14.48 | -6.10 |
| 133 | 20.59 | 17.65 | -2.94 |
| 134 | 20.63 | 16.57 | -4.06 |
| 135 | 20.74 | 15.05 | -5.69 |
| 136 | 20.87 | 15.98 | -4.89 |
| 137 | 21.04 | 15.23 | -5.81 |
| 138 | 21.13 | 16.09 | -5.04 |
| 139 | 21.21 | 15.86 | -5.35 |
| 140 | 21.24 | 16.76 | -4.48 |
| 141 | 21.25 | 17.01 | -4.24 |
| 142 | 21.28 | 34.37 | +13.09 |
| 143 | 21.39 | 18.90 | -2.49 |
| 144 | 21.42 | 29.92 | +8.50 |
| 145 | 21.47 | 16.17 | -5.30 |
| 146 | 21.48 | 17.78 | -3.70 |
| 147 | 21.71 | 16.93 | -4.78 |
| 148 | 21.73 | 17.88 | -3.85 |
| 149 | 22.02 | 17.52 | -4.50 |
| 150 | 22.06 | 13.41 | -8.65 |
| 151 | 22.14 | 24.24 | +2.10 |
| 152 | 22.16 | 20.58 | -1.58 |
| 153 | 22.38 | 13.17 | -9.21 |
| 154 | 22.38 | 18.53 | -3.85 |
| 155 | 22.47 | 14.62 | -7.85 |
| 156 | 22.49 | 17.96 | -4.53 |
| 157 | 22.54 | 17.62 | -4.92 |
| 158 | 22.55 | 18.66 | -3.89 |
| 159 | 22.97 | 15.02 | -7.95 |
| 160 | 23.11 | 22.77 | -0.34 |
| 161 | 23.12 | 19.33 | -3.79 |
| 162 | 23.31 | 37.28 | +13.97 |
| 163 | 23.37 | 17.03 | -6.34 |
| 164 | 23.66 | 13.55 | -10.11 |
| 165 | 23.77 | 17.80 | -5.97 |
| 166 | 24.26 | 13.11 | -11.15 |
| 167 | 24.70 | 16.09 | -8.61 |
| 168 | 25.24 | 16.31 | -8.93 |
| 169 | 25.65 | 19.33 | -6.32 |
| 170 | 25.76 | 14.62 | -11.14 |
| 171 | 26.10 | 26.51 | +0.41 |
| 172 | 26.33 | 20.66 | -5.67 |
| 173 | 26.53 | 24.65 | -1.88 |
| 174 | 26.69 | 16.54 | -10.15 |
| 175 | 27.02 | 18.46 | -8.56 |
| 176 | 27.05 | 19.24 | -7.81 |
| 177 | 27.22 | 17.59 | -9.63 |
| 178 | 27.68 | 16.40 | -11.28 |
| 179 | 28.01 | 20.82 | -7.19 |
| 180 | 30.47 | 72.79 | +42.32 |
| 181 | 30.94 | 19.30 | -11.64 |
| 182 | 31.91 | 14.17 | -17.74 |
| 183 | 32.25 | 18.32 | -13.93 |
| 184 | 33.15 | 16.47 | -16.68 |
Reading the chart
Each dot is one case. On the line, the model and the surgeon agree.
How often it is close
67 of 184 cases are within 2 mm, 94 within 3 mm and 127 within 5 mm.
Where it goes wrong
It pulls answers toward the middle. Below 15 mm it reads about 2 mm high. Above 25 mm it reads about 6 mm low, which is why it caught only 2 of the 17 cases the surgeon measured above 25 mm.
The points themselves
The tip is typically 1.9 px from the surgeon’s mark and the apex 4.1 px. The measured width is within 10% of the surgeon’s on 305 of 368 X-rays.
What we tried
We asked an AI to read the distance
Its answers stayed close to the 25 mm written in the question.
9.38
20 cases
We asked it for pixel positions instead
Better, but not repeatable. The same 10 cases scored 3.73 mm, then 5.60 mm on a second run.
3.73
10 cases
We trained our own model on the surgeon's marks
It found the tip and apex, but not the width of the screw.
109
159 cases
We measured the width from the image
Tested on crops drawn around the surgeon's marks, which a new case does not have.
3.40
185 cases
We let it find the screw on its own
The whole photo goes in with no help. This is what runs today.
4.31
184 cases
Rows 01 and 02 were tested on few cases because each run cost money. Row 04 scores better than row 05 because it was helped by the surgeon's marks.
Limits
Make the 25 mm call
It caught 2 of the 17 cases the surgeon measured above 25 mm. It should not be used to decide on its own.
Tell us how much surgeons disagree
One surgeon marked every case, so there is no second reading to compare against yet.
Get past the photo
In a phone photo of the monitor the screw is only about 20 pixels wide. A photo zoomed in on the hip would help every method, including a person.
In a typical photo the tip and apex are about 19 pixels apart. If every point were off by 2 pixels, even a perfect reader would be about 1.8 mm out on average.
Technical details
The scores above come from full-size models, each tested only on cases it did not train on. The server runs the smaller compressed copies to fit in memory. Those were checked on 6 cases, not on all 184.
Author
Julien Hosri
Product designer and engineer