{"id":9905,"date":"2022-09-21T08:35:06","date_gmt":"2022-09-21T03:05:06","guid":{"rendered":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/"},"modified":"2022-09-21T08:35:06","modified_gmt":"2022-09-21T03:05:06","slug":"solved-how-to-make-an-api-call-faster-in-golang","status":"publish","type":"post","link":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/","title":{"rendered":"[Solved] How to make an api call faster in Golang?"},"content":{"rendered":"<p> [ad_1]<br \/>\n<\/p>\n<div id=\"answer-70218911\" class=\"answer js-answer accepted-answer js-accepted-answer\" data-answerid=\"70218911\" data-parentid=\"70217232\" data-score=\"1\" data-position-on-page=\"1\" data-highest-scored=\"1\" data-question-has-accepted-highest-score=\"1\" itemprop=\"acceptedAnswer\" itemscope itemtype=\"https:\/\/schema.org\/Answer\">\n<div class=\"post-layout\">\n<div class=\"votecell post-layout--left\"><\/div>\n<div class=\"answercell post-layout--right\">\n<div class=\"s-prose js-post-body\" itemprop=\"text\">\n<p>Consider a worker pool pattern like this: <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/go.dev\/play\/p\/p6SErj3L6Yc\">https:\/\/go.dev\/play\/p\/p6SErj3L6Yc<\/a><\/p>\n<p>In this example application, I&#8217;ve taken out the API call and just list the file names.  That makes it work on the playground.<\/p>\n<ul>\n<li>A fixed number of worker goroutines  are started.  We&#8217;ll use a channel to distribute their work and we&#8217;ll close the channel to communicate the end of the work.  This number could be 1 or 1000 routines, or more.  The number should be chosen based on how many concurrent API operations your putio API can reasonably be expected to support.<\/li>\n<li><code>paths<\/code> is a <code>chan string<\/code> we&#8217;ll use for this purpose.<\/li>\n<li>workers <code>range<\/code> over <code>paths<\/code> channel to receive new file paths to upload<\/li>\n<\/ul>\n<pre><code>package main\n\nimport (\n    \"fmt\"\n    \"os\"\n    \"path\/filepath\"\n    \"sync\"\n)\n\nfunc main() {\n    paths := make(chan string)\n    var wg = new(sync.WaitGroup)\n    for i := 0; i &lt; 10; i++ {\n        wg.Add(1)\n        go worker(paths, wg)\n    }\n    if err := filepath.Walk(\"\/usr\", func(path string, info os.FileInfo, err error) error {\n        if err != nil {\n            return fmt.Errorf(\"Failed to walk directory: %T %w\", err, err)\n        }\n        if info.IsDir() {\n            return nil\n        }\n        paths &lt;- path\n        return nil\n    }); err != nil {\n        panic(fmt.Errorf(\"failed Walk: %w\", err))\n    }\n    close(paths)\n    wg.Wait()\n}\n\nfunc worker(paths &lt;-chan string, wg *sync.WaitGroup) {\n    defer wg.Done()\n    for path := range paths {\n        \/\/ do upload.\n        fmt.Println(path)\n    }\n}\n<\/code><\/pre>\n<p>This pattern can handle an indefinitely large amount of files without having to load the entire list in memory <em>before<\/em> processing it.  As you can see, this doesn&#8217;t make the code more complicated &#8211; actually, it&#8217;s simpler.<\/p>\n<blockquote>\n<p>When I run the program it just uploads one file which is the one<\/p>\n<\/blockquote>\n<p>Function literals inherit the scope in which they were defined.  This is why our code only listed one path &#8211; the <code>path<\/code> variable scope in the for loop was shared to each go routine, so when that variable changed, all routines picked up the change.<\/p>\n<p>Avoid function literals unless you actually <em>want<\/em> to inherit scope.  Functions defined at the global scope don&#8217;t inherit any scope, and you must pass all relevant variables to those functions instead.  This is a good thing &#8211; it makes the functions more straightforward to understand and makes variable &#8220;ownership&#8221; transitions more explicit.<\/p>\n<p>An appropriate case to use a function literal could be for the <code>os.Walk<\/code> parameter; its arguments are defined by <code>os.Walk<\/code> so definition scope is one way to access other values &#8211; such as <code>paths<\/code> channel, in our case.<\/p>\n<p>Speaking of scope, global variables should be avoided unless their scope of usage is truly global.  Prefer passing variables between functions to sharing global variables.  Again, this makes variable ownership explicit and makes it easy to understand  which functions do and don&#8217;t access which variables.  Neither your wait group nor your <code>filePaths<\/code> have any cause to be global.<\/p>\n<pre><code>            f, err := os.Open(path)\n<\/code><\/pre>\n<p>Don&#8217;t forget to close any files you open.  When you&#8217;re dealing with 40 or 50 files, letting all those open file handles pile up until the program ends isn&#8217;t so bad, but it&#8217;s a time bomb in your program that will go off when the number of files exceeds the <code>ulimit<\/code> of allowed open files.  Because the function execution greatly exceeds the part where the file needs to be open, <code>defer<\/code> doesn&#8217;t make sense in this case.   I would use an explicit <code>f.Close()<\/code> after uploading the file.<\/p>\n<\/p><\/div>\n<div class=\"mt24\"><\/div>\n<\/div>\n<p>            <span class=\"d-none\" itemprop=\"commentCount\">5<\/span> <\/p><\/div>\n<\/div>\n<p>[ad_2]<\/p>\n<p>solved How to make an api call faster in Golang? <\/p>\n","protected":false},"excerpt":{"rendered":"<p>[ad_1] Consider a worker pool pattern like this: https:\/\/go.dev\/play\/p\/p6SErj3L6Yc In this example application, I&#8217;ve taken out the API call and just list the file names. That makes it work on the playground. A fixed number of worker goroutines are started. We&#8217;ll use a channel to distribute their work and we&#8217;ll close the channel to communicate &#8230; <a title=\"[Solved] How to make an api call faster in Golang?\" class=\"read-more\" href=\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/\" aria-label=\"More on [Solved] How to make an api call faster in Golang?\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[320],"tags":[1315,929,596,2187],"class_list":["post-9905","post","type-post","status-publish","format-standard","hentry","category-solved","tag-api","tag-concurrency","tag-go","tag-goroutine"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>[Solved] How to make an api call faster in Golang? - JassWeb<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"[Solved] How to make an api call faster in Golang? - JassWeb\" \/>\n<meta property=\"og:description\" content=\"[ad_1] Consider a worker pool pattern like this: https:\/\/go.dev\/play\/p\/p6SErj3L6Yc In this example application, I&#8217;ve taken out the API call and just list the file names. That makes it work on the playground. A fixed number of worker goroutines are started. We&#8217;ll use a channel to distribute their work and we&#8217;ll close the channel to communicate ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/\" \/>\n<meta property=\"og:site_name\" content=\"JassWeb\" \/>\n<meta property=\"article:published_time\" content=\"2022-09-21T03:05:06+00:00\" \/>\n<meta name=\"author\" content=\"Kirat\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kirat\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/\"},\"author\":{\"name\":\"Kirat\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\"},\"headline\":\"[Solved] How to make an api call faster in Golang?\",\"datePublished\":\"2022-09-21T03:05:06+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/\"},\"wordCount\":469,\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"keywords\":[\"api\",\"concurrency\",\"go\",\"goroutine\"],\"articleSection\":[\"Solved\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/\",\"url\":\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/\",\"name\":\"[Solved] How to make an api call faster in Golang? - JassWeb\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#website\"},\"datePublished\":\"2022-09-21T03:05:06+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/jassweb.com\/solved\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"[Solved] How to make an api call faster in Golang?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/jassweb.com\/solved\/#website\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"name\":\"JassWeb\",\"description\":\"Build High-quality Websites\",\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/jassweb.com\/solved\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\",\"name\":\"Jass Web\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"contentUrl\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"width\":693,\"height\":132,\"caption\":\"Jass Web\"},\"image\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\",\"name\":\"Kirat\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"contentUrl\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"caption\":\"Kirat\"},\"sameAs\":[\"http:\/\/jassweb.com\"],\"url\":\"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"[Solved] How to make an api call faster in Golang? - JassWeb","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/","og_locale":"en_US","og_type":"article","og_title":"[Solved] How to make an api call faster in Golang? - JassWeb","og_description":"[ad_1] Consider a worker pool pattern like this: https:\/\/go.dev\/play\/p\/p6SErj3L6Yc In this example application, I&#8217;ve taken out the API call and just list the file names. That makes it work on the playground. A fixed number of worker goroutines are started. We&#8217;ll use a channel to distribute their work and we&#8217;ll close the channel to communicate ... Read more","og_url":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/","og_site_name":"JassWeb","article_published_time":"2022-09-21T03:05:06+00:00","author":"Kirat","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kirat","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/#article","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/"},"author":{"name":"Kirat","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31"},"headline":"[Solved] How to make an api call faster in Golang?","datePublished":"2022-09-21T03:05:06+00:00","mainEntityOfPage":{"@id":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/"},"wordCount":469,"publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"keywords":["api","concurrency","go","goroutine"],"articleSection":["Solved"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/","url":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/","name":"[Solved] How to make an api call faster in Golang? - JassWeb","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/#website"},"datePublished":"2022-09-21T03:05:06+00:00","breadcrumb":{"@id":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/jassweb.com\/solved\/solved-how-to-make-an-api-call-faster-in-golang\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/jassweb.com\/solved\/"},{"@type":"ListItem","position":2,"name":"[Solved] How to make an api call faster in Golang?"}]},{"@type":"WebSite","@id":"https:\/\/jassweb.com\/solved\/#website","url":"https:\/\/jassweb.com\/solved\/","name":"JassWeb","description":"Build High-quality Websites","publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/jassweb.com\/solved\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/jassweb.com\/solved\/#organization","name":"Jass Web","url":"https:\/\/jassweb.com\/solved\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/","url":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","contentUrl":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","width":693,"height":132,"caption":"Jass Web"},"image":{"@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31","name":"Kirat","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/","url":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","contentUrl":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","caption":"Kirat"},"sameAs":["http:\/\/jassweb.com"],"url":"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/9905","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/comments?post=9905"}],"version-history":[{"count":0,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/9905\/revisions"}],"wp:attachment":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/media?parent=9905"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/categories?post=9905"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/tags?post=9905"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}