next statement in function to go next in loop where function is used

I made a function to handle errors when webscraping:

get_page <- function(url, user_agent = "my information", skipped) {
  
  headers <- add_headers(`User-Agent` = user_agent)
  
  if (!exists(skipped, envir = .GlobalEnv)) {
    assign(skipped, c(), envir = .GlobalEnv)
  }
  
  for (attempt in 1:4) {
    response <- GET(url, headers = headers)
    http_code <- response$status_code
    
    if (http_code == 200) {
      page <- read_html(rawToChar(response$content))
      return(page)
    } else if (http_code %in% c(400, 401, 403, 404)) {
      cat("HTTP", http_code, "error for", url, "\n")
      assign(skipped, c(get(skipped), url), envir = .GlobalEnv)
      return(NULL)
    }
    
    print(http_code)
    
    if (attempt == 4) {
      cat("Skipped", url, "\n")
      assign(skipped, c(get(skipped), url), envir = .GlobalEnv)
      return(NULL)
    }
    
    Sys.sleep(30)
  }
}

Currently, if it returns error codes 400-404 or if it fails 3 times, I handle this by checking if page is NULL after I use the function:

for (url in urls){ 
  page <- get_page(url, skipped = "skipped_pers_aau")
  
  if (is.null(page)) {
    next
  }

  #...here comes the parsing and storing part

However, this means that i have to include the if (is.null(page))-part after everytime I call the get_page function. It would be better if the function was able to go to the next url in the urls itself, however, to my knowledge, if i use next within a function, this won’t go to the next part of the loop where the function is used inside.

Are there any way I can modify the function, so I could just do the webscraping and error handling with:

for (url in urls){ 
  page <- get_page(url, skipped = "skipped_pers_aau")
  
  #...here comes the parsing and storing part

…i.e., let the function know that it should go to the next part of the loop where the function is used inside (not having to include the if (is.null(page))-part or anything else, but simple just define page using the function, and then this alone would handle the errors)?

>Solution :

As @r2evans said in the comments, you cannot pass next from a function. However, you can refactor. As well as your function returning NULL, have it return an error. We can use tryCatch() to make the loop continue after the error.

I am just going to use toy functions to demonstrate the concept. The urls will be c("a", "b", "c", "d", "e"), and "c" will return an error:

get_page <- function(url) {
    if (url == "c") stop("Error") else url
}

We can also create a parse_page() function which returns NULL if it is supplied a page which caused an error.

parse_page <- function(page) {
    if (is.null(page)) {
        return(NULL)
    }
    # Do the parsing here but in this
    # case we'll return the whole page
    page
}

Technically this does not require you to raise an error, as it just checks for NULL (which stop() returns), but I think it’s better to do raise an error (or at least a warning) if you plan to use this function in other places.

Then your loop might look something like:

urls <- letters[1:5]
page_list <- vector(mode = "list", length = length(urls)) |>
    setNames(urls)
for (url in urls) {
    page <- tryCatch(get_page(url), error = \(e) message(e, "\n"))

    page_list[[url]] <- parse_page(page)
}

This will print your error message but it will keep going. Your page_list object will be:

page_list
# $a
# [1] "a"

# $b
# [1] "b"

# $d
# [1] "d"

# $e
# [1] "e"