How to get content from a url in Go

Reading from a url resource in Go
Created
Modified

Fetching a URL

Go provides a collection of packages, grouped under net, that make it easy to send and receive information through the Internet. Package http provides HTTP client and server implementations.

package main

import (
  "io/ioutil"
  "net/http"
)

func main() {

  url := "https://go.dev/"

  resp, err := http.Get(url)
  if err != nil {
    // panic
  }
  defer resp.Body.Close()
  b, err := ioutil.ReadAll(resp.Body)

  // upload image
  resp, err := http.Post("http://example.com/upload", "image/jpeg", &buf)

  // post form
  resp, err := http.PostForm("http://example.com/form", url.Values{"key": {"Value"}, "id": {"123"}})

}

The client must close the response body when finished with it.

For control over HTTP client headers

client := &http.Client{
  CheckRedirect: redirectPolicyFunc,
}

resp, err := client.Get("http://example.com")
// ...

req, err := http.NewRequest("GET", "http://example.com", nil)
// ...
req.Header.Add("If-None-Match", `W/"wyzzy"`)
resp, err := client.Do(req)
// ...

// For control over proxies ...
tr := &http.Transport{
  MaxIdleConns:       10,
  IdleConnTimeout:    30 * time.Second,
  DisableCompression: true,
}
client := &http.Client{Transport: tr}
resp, err := client.Get("https://example.com")

Fetching URLs Concurrently

One of the most interesting and novel aspects of Go is its support for concurrent program- ming.

package main

import (
  "fmt"
  "io"
  "io/ioutil"
  "net/http"
  "time"
)

func fetch(url string, ch chan<- string) {
  start := time.Now()
  resp, err := http.Get(url)
  if err != nil {
    ch <- fmt.Sprint(err) // send to channel ch
    return
  }
  defer resp.Body.Close()

  nbytes, err := io.Copy(ioutil.Discard, resp.Body)
  if err != nil {
    ch <- fmt.Sprintf("while reading %s: %v", url, err)
    return
  }
  secs := time.Since(start).Seconds()
  ch <- fmt.Sprintf("%.2fs  %7d  %s", secs, nbytes, url)
}

func main() {

  urls := []string{"https://go.dev/", "https://www.google.com/"}

  ch := make(chan string)
  for _, url := range urls {
    go fetch(url, ch)
  }

  for range urls {
    fmt.Println(<-ch) // receive from channel ch
  }

}

0.95s    15227  https://www.google.com/
1.13s    58557  https://go.dev/

Questions

How to read gzipped data from http response in Golang?

Golang by default will automatically decode the body of gzipped response. If the Transport requests gzip on its own and gets a gzipped response, it's transparently decoded in the Response.Body. However, if the user explicitly requested gzip it is not automatically uncompressed.

Related Tags

#fetch# #url# #http.Get#

MORE FROM THE INSTALLMD