https://blog.maxgio.me/categories/go/ Recent content in go on Maxgio's blog https://blog.maxgio.me/papermod-cover.png https://blog.maxgio.me/papermod-cover.png Hugo -- gohugo.io Mon, 04 Sep 2023 10:20:00 +0200 https://blog.maxgio.me/posts/improving-consistency-performance-go-crawler-retry-logics-http-client-tuning/ Mon, 04 Sep 2023 10:20:00 +0200 https://blog.maxgio.me/posts/improving-consistency-performance-go-crawler-retry-logics-http-client-tuning/ Introduction wfind is a simple web crawler for files and folders in web pages hyerarchies. The goal is basically the same of GNU find for file systems. At the same time it’s inspired by GNU wget, and it merges the find features applied to files and directories exposed as HTML web resources. In this blog we’ll go through the way I improved consistency in this crawler, by implementing retry logics and tuning network and transport in the HTTP client.