That code did not work for me, but I search in better amp plugin for a "better" function to call and I change your code, and this work for me:
add_action( 'rank_math/frontend/canonical', function( $url ) {
if ( ! is_singular() || ! is_category() || ! function_exists( 'better_amp_get_canonical_url' ) ) {
$url = better_amp_guess_none_amp_url();
return $url;
}
$custom_url = RankMath\Post::get_meta( 'canonical_url', get_the_ID() );
if ( $custom_url ) {
return $custom_url;
}
return better_amp_get_canonical_url();
});