avoiding abort in libgmp-Collection of common programming errors

The best way to handle these errors gracefully in your application is probably to fork off a helper process to perform the GMP calculations. If the helper process is killed by SIGABRT, your parent process can detect that and report an error to the user.

(The below is my original answer, which has “undefined results” according to the GMP documentation – it is left here for completeness).

You can catch the error if you install a signal handler for SIGABRT that uses longjmp():

jmp_buf abort_jb;

void abort_handler(int x)
{
    longjmp(abort_jb, 1);
}

int dofac(unsigned long n)
{
    signal(SIGABRT, abort_handler);
    if (setjmp(abort_jb))
        goto error;

    mpz_t result;
    mpz_init(result);

    mpz_fac_ui(result, 20922789888000);

    std::cout