Rendered at 10:36:23 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
aleks_me2 14 hours ago [-]
I've revived AWFFull for static log analysis, just like in the good old webalizer days :-). Link to the report is in my bio, sorry, I can't post links here for now.
What I'm seeing matches the observations of the other commenters here exactly: most of the "users" are bots harvesting pages and PDFs. My solution for getting at the real users is GoatCounter.
It is now mostly unusable for web analytics. So many scrapers are pretending to be browsers. On my site, there is a factor 10 with what I consider an honest view of human visitors.
herbst 2 hours ago [-]
When customers ask me for visitor metrics I tend to explain that it doesn't matter anymore before I show them them the highly inflated absolutely unrealistic numbers.
joshyi 11 hours ago [-]
If you're using it to serve the marketing team, then yeah. We use it more for server performance and metrics that cookie-cutter JS analytics tools can't really provide.
btw, JS-based analytics are no longer as accurate for identifying "real humans" as they used to be. bots have gotten much better at behaving like a person behind a keyboard, so it's increasingly easy for them to make your analytics tool think they're a real user
janvdberg 16 hours ago [-]
Yes, this is unfortunately my experience with GoAccess
iJohnDoe 14 hours ago [-]
This always looks cool and the real-time aspect always gets attention. However, If I remember correctly, it was kind of a burden to get deployed.
Again, if I remember correctly, it was able to make reports for AWS Cloudfront logs, which was actually useful.
QuantumNomad_ 13 hours ago [-]
> If I remember correctly, it was kind of a burden to get deployed
It probably depends a lot on your setup. I’ve been running GoAccess from a crontab on some of my sites for a couple of years. It was a bit fiddly to get the correct combination of Nginx logs, forwarding real IP from HAProxy, and getting it to show with GeoIP in generated GoAccess HTML reports, but once I got it working satisfactorily it’s been working well since. The only part that is missing in my GoAccess reports is the ASN of the clients. I don’t remember if that’s because ASN mappings are a paid GeoIP database feature or if I simply ignored that part or what, but I do have countries and municipalities correct (I can tell because for example on one of the sites the most frequent visitor is a client machine that I own, and it shows up accordingly prominently with the country and municipality where that client machine is located) and that’s the main thing I wanted to be able to see when setting up GoAccess to use a GeoIP database file.
I serve the generated GoAccess HTML reports from Nginx itself with a simple .htaccess file providing username and hashed password and Nginx using that file with HTTP Basic Auth. Since the only person that looks at those reports is myself, and the sites use TLS, I don’t need anything more advanced than HTTP Basic Auth to protect access to the GoAccess HTML reports.
And I suppose that in today’s world an LLM would probably be able to do most of the config writing for a setup such as mine for all of the above parts.
aleks_me2 2 hours ago [-]
The question "which ASN are the clients coming from?" was exactly the reason I revived AWFFull — the new version adds an ASN table based on the DB-IP ASN Lite database :-)
To your point about it maybe being a paid feature: the DB-IP Lite files are free (CC-BY), so the ASN part doesn't have to cost anything — it's just not what the MaxMind-oriented tooling usually reaches for.
QuantumNomad_ 2 hours ago [-]
Sweet! After reading your reply I had a look and downloaded the DB-IP ASN Lite db also. From before I had the DB-IP City Lite db. And then making GoAccess use the ASN db was just a matter of adding the same --geoip-database argument again to GoAccess that I already use with the city db file path, but with the ASN db file path. As per the manual https://goaccess.io/man which even includes exactly using both city and ASN db together as an example :D
This is why it’s so great to share experiences on HN :) I wasn’t really planning on looking further into how to set up the ASN part but then after your comment I basically had no choice other than to have a look to see if the ASN Lite db file was the missing part and it was :D Thank you :)
aleks_me2 13 hours ago [-]
The question I always come back to is: how long does it take you to get stats for something like 1 month or 1 quarter with the current "modern" tools?
For ad hoc analytics these tools are very nice, but for long-term stats they run into timeouts, because the backend needs a lot of time to collect the data and build the reports for ~1 month or a quarter — or am I wrong about that?
joshyi 11 hours ago [-]
It's a oneliner for us, we run Caddy at the office and just specify --log-format=CADDY --keep-last=90 --date-spec=min and a single instance is ingesting ~100M/req a month
WesolyKubeczek 15 hours ago [-]
Unfortunately, each time a new log analyzer appears, it seems to only be analyzing web access logs in combined format.
tredre3 13 hours ago [-]
GoAccess supports more than combined format. I think they even support stubborn servers who output JSON logs only:
and honestly, there's still no other tool that does what GoAccess does quite as well. Most of the alternatives are cookie-cutter, JS-based Google Analytics wannabes. And with bots becoming much smarter and more prevalent when accessing websites, relying solely on JS-based tracking isn't as trustworthy anymore. We use GoAccess to capture what JS-based analytics can't, e.g., server side perf and traffic data. It's not meant to serve the marketing team though.
What I'm seeing matches the observations of the other commenters here exactly: most of the "users" are bots harvesting pages and PDFs. My solution for getting at the real users is GoatCounter.
btw, JS-based analytics are no longer as accurate for identifying "real humans" as they used to be. bots have gotten much better at behaving like a person behind a keyboard, so it's increasingly easy for them to make your analytics tool think they're a real user
Again, if I remember correctly, it was able to make reports for AWS Cloudfront logs, which was actually useful.
It probably depends a lot on your setup. I’ve been running GoAccess from a crontab on some of my sites for a couple of years. It was a bit fiddly to get the correct combination of Nginx logs, forwarding real IP from HAProxy, and getting it to show with GeoIP in generated GoAccess HTML reports, but once I got it working satisfactorily it’s been working well since. The only part that is missing in my GoAccess reports is the ASN of the clients. I don’t remember if that’s because ASN mappings are a paid GeoIP database feature or if I simply ignored that part or what, but I do have countries and municipalities correct (I can tell because for example on one of the sites the most frequent visitor is a client machine that I own, and it shows up accordingly prominently with the country and municipality where that client machine is located) and that’s the main thing I wanted to be able to see when setting up GoAccess to use a GeoIP database file.
I serve the generated GoAccess HTML reports from Nginx itself with a simple .htaccess file providing username and hashed password and Nginx using that file with HTTP Basic Auth. Since the only person that looks at those reports is myself, and the sites use TLS, I don’t need anything more advanced than HTTP Basic Auth to protect access to the GoAccess HTML reports.
And I suppose that in today’s world an LLM would probably be able to do most of the config writing for a setup such as mine for all of the above parts.
To your point about it maybe being a paid feature: the DB-IP Lite files are free (CC-BY), so the ASN part doesn't have to cost anything — it's just not what the MaxMind-oriented tooling usually reaches for.
This is why it’s so great to share experiences on HN :) I wasn’t really planning on looking further into how to set up the ASN part but then after your comment I basically had no choice other than to have a look to see if the ASN Lite db file was the missing part and it was :D Thank you :)
For ad hoc analytics these tools are very nice, but for long-term stats they run into timeouts, because the backend needs a lot of time to collect the data and build the reports for ~1 month or a quarter — or am I wrong about that?
https://goaccess.io/man#custom-log