We talked about this during the outage before but any API requests related to account information should not be going through a cacheable CDN.
At the mild end, you could get stale data. Deposit some money and the cache doesn't reflect making you wonder whether the deposit happened or not.
On the worst end, you get this - all users get exposed the most recently requested data.
All this to save... a few kB?
Whether a CDN is caching or not could itself just be a misconfiguration.
Passing dynamic non-cached requests through a CDN isnât about saving a few kB for the server (by definition it still has to generate it). Itâs about improving end-user performance through things like having edge termination of TLS.
So itâs not so easy to have such a blanket statement as no API requests should go through a CDN. While itâs all too easy to inadvertently turn on the caching of requests.
> Itâs about improving end-user performance through things like having edge termination of TLS.
Err, wouldn't the CDN itself still need to do the same TLS handsake with the end server?
Most CDNs out there maintain a pool of HTTPS keep-alive'd connections to backing services. This eliminates the roundtrip needed for establishing a TCP connection and the roundtrip (or two) needed for setting up TLS.
Another reason a no-caching CDN can help is when that CDN has a higher quality connection to the backing service than the user has. A lot of eyeball networks out there might have terrible transit connectivity to the global Internet during peak hours (because that cost money), but generally have high quality settlement-free peering with CDNs. By paying a CDN, you're effectively paying them to provide better Internet connectivity to your users.
In my case I use Cloudflare Tunnel [0] which is an actual agent installed on my web servers that creates a reverse tunnel back to Cloudflare's network.
This creates a complete disconnection between whatever TLS handshakes are happening between the client and my CDN (Cloudflare) and whatever's going on locally between the tunnel daemon and my local web server process.
For example you can choose to run these tunnels without HTTPS at all on your local webserver (bearing in mind that you have no inbound firewall ports open to the webserver box - no traffic leaves that box unencrypted), the daemon directly proxies your local-only HTTP port on the machine over an encrypted tunnel to Cloudflare and via whatever flavour of Cloudflare TLS you like to your end-users.
CDN edge servers maintain a warm pool of TCP/TLS connections to origin servers, which eliminates the need for a handshake and significantly improves performance.
I'm not an expert, but if the TLS handshake occurs closer (as in latency) to the client and the actual request/response payload is small (or if the CDN's network is better), then it seems like terminating in a CDN could certainly be better than not.
And CDN's offer flood / ddos protection as well.
I agree.
I've run some eCommerce sites before, we were PCI Tier 1, roughly 1% of web traffic: but nowhere _near_ the level of security Klarna should have; we had the CDN's activated only on subdomains... always.
Usually static.<site>.com or whatever..
This has the added benefit of not sending cookies to the CDN, because that can affect latency and is also a bit less secure because a static image/css/js does not need the cookie. (and bandwidth, before we were charged per byte (and that's egress now anyway in most cases, god I'm old.)).
If I had to venture a guess, they're probably scared of getting slammed with a DDoS so they "guard" everything behind the CDN, it's not about caching.
We got slammed with DDoS's too so we had to have systems in place to push BGP routes via verisign to "scrub" the traffic... we didn't even have that many "credits" and it cost millions...
I suspect the background is the app is showing html-views rendered on the server, and they are sharing as much infrastructure as possible. To make it easier on the app team.
If they separated app/cdn/web with different SSL certificates, something like this wouldn't be possible even when developers eff up.
> We talked about this during the outage before but any API requests related to account information should not be going through a cacheable CDN.
That is simply not correct.
The entire reason why Fastly is so popular is because it lets you cache the API using cache-bust on write. Between the request collapsing, shielding, and a near immediate cache invalidation caching API allows one to accept enormous spikes of traffic that normally would bring down the origins.
Unrelated to the incident, but to Klarna.
What is Klarna _for_? What does Klarna do that others don't? They brand themselves as "We enable you to buy hip shit you don't have the money for [and will become irrelevant before you payed off your debt]". Almost literally [0]
Is it more of the same but dressed pretty or am I missing something?
That said, sometimes a common thing done up well will be successful. Looking at you, Slack.
In Europe, strong customer authentication is now required when making credit card purchases https://en.wikipedia.org/wiki/Strong_customer_authentication
So Klarna delays having to authenticate with the bank which issued your credit card from the time of purchase, to the time of paying your Klarna "credit card bill".
They also delay sending your invoice until the item is actually shipped, and you then have a 14 day due date. This means returns and refunds have no money exchanged from any of your own accounts, so you don't have to worry about how long it takes for the refund to enter your credit card.
Edit: They also remember you address based on your email and name, across any vendors, so check out is really quick even if it is the first time you shop at a specific site. This is of course great for smaller vendors.
Klarna in Europe I think for most people I know is a "i made this mistake once" experience. I know nobody who picked the Klarna transaction type more than once.
The checkout is super quick only to be frustrated with the payment process later. And since it's Klarna that comes after you to collect payment it can eventually backfire on the business that you ordered from as well.
What's wrong with the payment process in your country? Where I am it's as easy as paying any invoice I get from a credit card company or I can by with debit card.
Also in my country, in Europe, Klarna is mentioned more and more in news articles about consumer debt and predatory lending, so I think their market share among people who keep credit card balances is increasing. They have stopped or kept pending unusually large purchases by me pending a credit check, but they could probably do a lot more to not get a reputation as lumped together with the most predatory consumer loan banks.
Well, they are all in one payment processor, loan institute and debt collector with terrible record, with some questionable business practices.
They are an elaborate pay day loans company.
Yeah, I would agree with that: it's payday loans with a friendly face. I once worked with someone who had previously been an exec at Klarna, and he had some 'interesting' stories about how they did credit decisioning. One of the examples that always stuck in my mind[0] was declining people with five or more vowels in their last name, because it was an effective proxy for West African people. (I remember that one because it also picks up my somewhat posh double-barrelled English surname, so I suppose I would have been collateral damage.)
[0] I should say I can't remember for sure if this was from his time at Klarna or somewhere else, but it was at least illustrative of the kind of people and the culture there.
Their main selling point is offering high interestrate credit to sites customers and paying the site the money directly, taking the risk.
It's funny that I haven't heard about it before, although I live in Germany and Klarna is probably the biggest payment processor in Germany.
Luckily, I haven't used it. I never liked that one need to provide his bank login data to Klarna / Sofort to use it. How is that even legal?
EDIT: I haven't heard about Klarna outage, Klarna is well known.
> I never liked that one need to provide his bank login data to Klarna / Sofort to use it. How is that even legal?
I am parroting other HN comments here, but the reason this is necessary is because banks do not provide a public (or private) API except between other banks. Therefore the only avenue for integration would be directly accessing the account.
Not really true in Germany where FinTS and its precursor HBCI have been supported by many banks since the 90s or so. Ironically, it's typically the hip start-up banks like N26 that don't support this standard.
I wish this wasn't normal to me, but I've been fully conditioned to expect this from QuickBooksOnline
They do though. PDS2 requires all banks to have an accessible API.
This is different from banks actually _having_ an API. Many have stalled or successfully asked for exemptions, and even those who have gone ahead and, to some extent, complied, have still tried to make their APIs as difficult to use, feature-poor, and unreliable as possible. Source: worked at a bank which actually tried to do this well, and which also offered functionality which required us to integrate with other banks' APIs.
> How is that even legal?
It's worse than that. Several banks tried to enforce in their terms and conditions that sharing your bank login data would be a breach of contract or at least absolve the bank of any liability. This was deemed in violation of antitrust laws for some reason, so now banks have to actively allow this which is just brain-dead IMHO.
The only upside of this is that we now have PSD2 now anyway, so the damage that can be done by someone who has your login credentials is limited.
> This was deemed in violation of antitrust laws for some reason
My best guess at what that "some reason" may be is that it breaks shoddily built integrations like [Teller](https://teller.io/), which, instead of using schemes like Open Banking where available, simply rely on credential sharing.
For others wondering what PSD2 is: https://en.wikipedia.org/wiki/Payment_Services_Directive and there were also a lot of "what is it and how does it affect you" articles from name brand bank websites, too, but I figured start here
In Finland it is illegal to give your bank login data to others, including scrapers. One couple both got sentenced when the other used the bank login data left on table to order something online. Fraud for the other and neglegence of leaving the login data easily accessible for the other.
I agree, havenât used it (knowingly) and regularly order from other sites if klarna is the only payment option.
Are you certain you havenât used it? Some shopping sites use them as payment processor for regular CC transactions without much visibility. You have to look at the T&C to even notice.
I usually don't leave CC info on some random site, only to major and well known. If the site is unknown, it's either PayPal, "pay by invoice" or no order at all.
Either PayPal or CC issuer provide at least some buyers' protection, while given your bank credentials to Klarna is normally against ToC of your bank.
I try not to use PayPal in the hope they will become irrelevant as a business (I know, not holding my breath), mainly because their customer service and fees are terrible.
CCs are not a great mechanism and I wished we had better bank services and safer integrations to process payments, but that's what we have. I check my bank statement regularly and I know I can get money back if fraud happens. If I don't trust a website I used a prepaid with the exact amount on.
> I usually don't leave CC info on some random site, only to major and well known.
That's the whole point of klarna (who also do CC processing so might have used them unknowingly): you don't need to trust Johnny's Little Online Store handling your CC, since the payment is handled by klarna.
This happened to Steam as well, where misconfigured caching leaked PII: https://store.steampowered.com/oldnews/19852
It used to happen even with everything configured correctly, if your users connected over plain HTTP, from an ISP that forced their traffic to go through a transparent proxy with overaggressive caching. It's not fun to hear about users logged in with somebody else's account, but only in remote corners of the world instead of the more densely populated areas as you'd expect.
The same happened to the Italian government: https://moondo.info/en/Luciano-Vangone-suddenly-the-most-fam...
Impressive success at burying the lead.
Let me fix for it.
"Detailed incident report: No One At Klarna Understands That Disabling CDN Cache Protections for Personal Information is Bad"
Their initial post blamed the problem on âhuman errorâ, an anti-pattern in getting to the root cause of things.
This post attempts to blame their CDN vendor.
âAll these reviewers couldnât spot it.â
The blog post doesn't say much more than the headline. I'm curious about the specifics of what could of actually happened here.
In my limited experience working with CDNs wouldn't you just cache the responses of unique URLs and have some sort of cookie check at the edge before serving it.
So my own app would request something like /api/account?id=123 with my own id in there.
How would you end up getting other people's data in your app if your app only calls that unique URL?
It's pretty easy to imagine an API having an endpoint like GET /api/account/mine, which is implicitly parameterised by the user ID associated with that session. Or even a 'list' rather than a 'lookup' endpoint, like GET /api/messages, which fetched all private messages associated with the authenticated user â or whatever the equivalent private information would be, in Klarna's 'domain'.
Edit: If the other commenter is correct, then it's less bad than I imagined. Or rather it would at least only be triggered, seemingly, if someone deliberately and maliciously requested something that didn't 'belong' to them.
I see, so the it's likely that a generic URL pattern (like your example) was accidentally included in their caching rules.
I guess I didn't think of that originally because I thought if you wanted to cache some kind of response data like this, why on earth would you use a generic URL? But perhaps they probably didn't intend to cache this endpoint.
> How would you end up getting other people's data in your app if your app only calls that unique URL?
From what I understand what happened with this outage, the CDN would still cache /api/account?id=123, and someone with account ID 234 could access it by altering the URL to retrieve the cached version, if account 123 has used the app recently.
That's because a CDN has (usually) no concept of authorization/authentication and can't make decisions that /api/account?id=123 shouldn't be served to someone other than the owner of account 123.
It would be less catastrophic (at least from a PR point of view) because people wouldn't get immediately served others' accounts, but you'd be vulnerable to attack.
In fact users did get other users' data just by using the app normally, so it likely didn't happen the way you describe it.
Right, what probably happened is that an `/accounts/me` endpoint got cached.
But what GP seems to be asking about is: âWould having your app always encode a user ID into the endpoint have helped?â.
What I'm wondering is: why would you ever want a CDN configuration to override no-cache instructions from the backend? I assume there's a use case for this, but I can't figure out what it is. Can anyone explain?
This sounds a bit like request collapsing on their CDN was enabled perhaps?
https://developer.fastly.com/learning/concepts/request-colla... fastly's VCL docs are interesting, particularly the bit about request queueing and `Cache-Control: private`. This sounds exactly like the kind of confusion that would lead to this kind of issue, while not being noticed in a low-traffic test environment with fewer simultaneous requests.
> HINT: One of many reasons you should be using Cache-Control and not Expires for your caching configuration is the ability to use directives like private. The private directive tells Fastly that this response is good for only one user, the one that originally triggered the request. Anyone else that got tagged onto the queue must now be dequeued and their requests processed individually.
> Responses not marked private, may be used to satisfy queued requests. Even a response with max-age=0 or the equivalent no-cache can be used to satisfy all the waiting clients, and the next request for the object will once again be a MISS. If your reason for giving a response a zero cache lifetime is that it contains content intended for a single user only, then ensure it is correctly marked private to avoid having this content sent to more than one user.
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.