Just happened to me with MongoDB. I don't mean the open source project, I mean the public company that sells premium SaaS hosted MongoDB (Atlas) instances along with their charts product.
I found a very serious bug that was trivially reproducible in my charts account (valid Mongo data being corrupted and displaying non-sensical data in charts). No way to report a bug, so I reached out through chat. Customer Support started playing the telephone game, asking me for various things every 2-3 weeks or so. If I didn't get back to them in 24 hours, I then got a warning that the bug would be closed out in another day.
This went on for several months. I sent them screenshot after screenshot along with the original mongoDB documents which showed very clearly they had a bug. Perhaps the fourth or fifth time that they asked me for some heavy lifting type deliverables, I didn't have time. They simply just closed the ticket.
Oh - near the end of that time, someone on their sales team cold-called me to try and upsell me on annual support contract. Yeah, right.
Anyway, the story has a happy ending. After getting the runaround for a few weeks, I created a Jupyter notebook that displayed the necessary chart I needed. It ended up being a lot more useful than the generic graphs I could pull from Atlas anyway.
It’s not just bug reports, companies won’t even tell you basic information about their services. Yesterday I hopped into online chat because I needed to find out the servers to add to our firewall exception list in order to prevent Spotify from playing. (We saw at least one web socket being blocked, but we needed to know what servers to unblock, we aren’t going to play whack-a-mole constantly)
They wouldn’t tell me. Eventually I asked Spotify to escalate to a supervisor, at least I did when they told me the only solution was to turn off our firewall!
The supervisor started asking what firewall we were using, whether we checked the firewall vendor’s community forums, etc.
I eventually said “look, to me this is not great. I work at a school with over 1,100 students, many of whom are paying for a Spotify subscription. We are happy to allow them to use your service and we are happy to add Spotify servers to our exception list, we just need to know what they are. If you decide not to tell us, we don’t mind - from my POV I’ve done my best to reach out to you and you have decided not to give me the information we need to allow your service.
I will therefore just tell all 1,100+ students that we reached out to Spotify but you guys refuse to tell us how to unblock your service. I’ll even show them this chat log. We aren’t going to disable our firewall and so I will just advise students to use a different paid service and that - until Spotify give us the basic info we need - Spotify won’t work.”
If a company like Spotify won’t even tell you what servers they use so we can allow access to them, then it’s no wonder stupid bugs like the one OP describes come about for years. I mean, Spotify is about to likely lose hundreds of subscriptions soon, even though we would prefer they wouldn’t, all because of some stupid policy that say that the servers they use to serve content to end users can’t be provided to network admins who want to allow their end users unfettered access to a service they pay for!
That's an unreasonable request for L1 support and an extremely uncommon request in 2023. I'm sure if you asked an SRE they could get you a list of IPs, but that list would change every day as new load balancers are deployed with new IPs and the old ones are shut down. IP+port firewalls are ancient tech that doesn't work with modern apps. Also what country is this? I've never seen a school (or any place with free wifi for that matter) whitelist apps like this in the US. They either allow everything or they only use a blacklist.
>Australia has some extremely strict filtering requirements
That's in line with all the horrible complaints I've heard about Australian internet.
>How does one get to an SRE at Spotify, out of interest?
noc@ email addresses are often unpublished, but silently open a ticket assigned to a network engineer. If they have any sense, they'll ignore your request for a list of IP addresses.
At the end of the day, Spotify isn’t going to work well at the school I work for. We filter for things like suicide attempts and to prevent access to adult content. We use an industry standard firewall and aren’t going to stop just because some random non-Australian thinks we don’t have a duty of care to our students.
We have had a lot of abuse in Australian schools. Almost every school has been affected by suicide and bullying. None of us want to go back to the way things were.
Yes. Ingress/egress IP ranges should be something support can easily provide.
Most places I've worked have had public-facing documentation that was a tiny bit dynamic (or at least had the one page updated by a cron) for the page containing the gateway IP lists or equivalent.
How does that work with third party CDNs, ipv6, dynamically allocated IPs from services like API gateway/ALBs, edge locations, geographic differences in content serving and the ability to evolve all of those without breaking an explicitly defined “these IPs are ours” contract for your consumer facing product that 99.9% of your users interact with through a standard connection and no weird firewall?
It worked quite well when I worked for a place that did this. I imagine the experience is quite similar elsewhere.
For CDNs, one of two things will be true: the traffic/access will be primarily API-related and not involve a CDN, or a CDN will be selected by the business which itself publishes its origin IP addresses, and the business publishing its IPs will either embed or link to that information. Cloudflare, for example, does this: https://www.cloudflare.com/ips/. Most good CDNs are popular enough that their IPs will already be allowed by most restrictive network environments.
IPv6 can be handled the same way. Just publish a list.
Dynamically allocated IPs for cloud provider infrastructure should be avoided by mature companies. Edge load balancers which do not support this should be avoided. All cloud providers make this easy: AWS, for example, provides a range of products that can enable this, from GA to NLB-with-ENI+EIP-in-front-of-ALB to NAT gateways to good old EIP'd EC2 instances routing traffic.
Handling multiple geos is as simple as publishing multiple lists of IPs. In the worst-case scenario, a customer uses the wrong geo's IP to allowlist traffic, things don't work, so they select a different one. If it's a frequent problem you can spend some time/money to provide alternative DNS records which always resolve to the same IPs and customers who need custom network configurations will endure a little extra latency.
> 99.9% of your users interact with through a standard connection and no weird firewall
That's doubtful, incomplete, and highly depends on the business you're in. Many companies which host user content, especially when they're new enough that they may not have grown content moderation/compliance procedures, end up on blocklists (which, like firewalls, are easiest to poke holes through on a per-IP basis). Several other things I've observed that complicate the situation further:
1. Some residential internet service providers firewall and/or block traffic, which can affect significant numbers of customers. This is an awful and borderline-unethical practice, but it happens.
2. Some very large corporations put all of their employees behind a firewall (via LAN or VPN). If you sell software to businesses, it is a big problem when a all of a business's users can't access your product while at work. Additionally, business infrastructure components reaching out to internet services to do API-API interactions will often need their traffic allowlisted by destination IP when leaving corporate infrastructure, and your gear will definitely need stable origin IPs if it ever initiates communication with business infrastructure.
3. Even if you're right and a statistically insignificant number of your users are thus affected, if you use dynamic IP addresses, that can change at any time. IPs associated with malicious traffic can be re-used by cloud providers. Even with non-reputationally-compromised IPs, for certain (usually non-HTTP) types of traffic, residential or corporate firewalls can make highly irrational/inconsistent decisions when deciding whether to allow traffic to an IP they haven't previously seen, even if the traffic in question is already common on the network. While such failures due to IP switching are rare, they're also very expensive: it really sucks to have to apologize to customers because you rotated out a load balancer or whatever.
Because it completely misses the point that it’s a lot of work that constricts future paths by forming a public contract that offers effectively no benefit to Spotify at all.
Those kids are going to pay for Spotify anyway regardless of if their weird school firewall comes up with a “this site is blocked” message or not. Phones and 5g exist, plus the school gets the ire for blocking it and not Spotify. And no residential ISP is ever going to block Spotify.
I found a very serious bug that was trivially reproducible in my charts account (valid Mongo data being corrupted and displaying non-sensical data in charts). No way to report a bug, so I reached out through chat. Customer Support started playing the telephone game, asking me for various things every 2-3 weeks or so. If I didn't get back to them in 24 hours, I then got a warning that the bug would be closed out in another day.
This went on for several months. I sent them screenshot after screenshot along with the original mongoDB documents which showed very clearly they had a bug. Perhaps the fourth or fifth time that they asked me for some heavy lifting type deliverables, I didn't have time. They simply just closed the ticket.
Oh - near the end of that time, someone on their sales team cold-called me to try and upsell me on annual support contract. Yeah, right.
Anyway, the story has a happy ending. After getting the runaround for a few weeks, I created a Jupyter notebook that displayed the necessary chart I needed. It ended up being a lot more useful than the generic graphs I could pull from Atlas anyway.