Hex digits are 4 bits and octal digits are 3 bits, so the digit boundaries don't line up. The reliable route goes through binary:
- Expand each digit into its bits: 4 per hex digit, 3 per octal digit.
- Regroup the bits from the right: in threes for octal, in fours for hex.
- Pad the leftmost group with zeros and read off the new digits.
You could go through decimal instead (convert to decimal, then repeated division), but that needs real arithmetic. The binary route is nothing but lookups (binary hex conversion, binary octal conversion).
What doesn't work is converting digit by digit. A5₁₆ is not "A → 12₈ and 5 → 5₈, so 125₈". That would be 85, but A5₁₆ is 165.