Hacker News
4 years ago by irjustin

We talked about this during the outage before but any API requests related to account information should not be going through a cacheable CDN.

At the mild end, you could get stale data. Deposit some money and the cache doesn't reflect making you wonder whether the deposit happened or not.

On the worst end, you get this - all users get exposed the most recently requested data.

All this to save... a few kB?

4 years ago by glenngillen

Whether a CDN is caching or not could itself just be a misconfiguration.

Passing dynamic non-cached requests through a CDN isn’t about saving a few kB for the server (by definition it still has to generate it). It’s about improving end-user performance through things like having edge termination of TLS.

So it’s not so easy to have such a blanket statement as no API requests should go through a CDN. While it’s all too easy to inadvertently turn on the caching of requests.

4 years ago by ownagefool

> It’s about improving end-user performance through things like having edge termination of TLS.

Err, wouldn't the CDN itself still need to do the same TLS handsake with the end server?

4 years ago by q3k

Most CDNs out there maintain a pool of HTTPS keep-alive'd connections to backing services. This eliminates the roundtrip needed for establishing a TCP connection and the roundtrip (or two) needed for setting up TLS.

Another reason a no-caching CDN can help is when that CDN has a higher quality connection to the backing service than the user has. A lot of eyeball networks out there might have terrible transit connectivity to the global Internet during peak hours (because that cost money), but generally have high quality settlement-free peering with CDNs. By paying a CDN, you're effectively paying them to provide better Internet connectivity to your users.

4 years ago by tailspin2019

In my case I use Cloudflare Tunnel [0] which is an actual agent installed on my web servers that creates a reverse tunnel back to Cloudflare's network.

This creates a complete disconnection between whatever TLS handshakes are happening between the client and my CDN (Cloudflare) and whatever's going on locally between the tunnel daemon and my local web server process.

For example you can choose to run these tunnels without HTTPS at all on your local webserver (bearing in mind that you have no inbound firewall ports open to the webserver box - no traffic leaves that box unencrypted), the daemon directly proxies your local-only HTTP port on the machine over an encrypted tunnel to Cloudflare and via whatever flavour of Cloudflare TLS you like to your end-users.

[0] https://www.cloudflare.com/en-gb/products/tunnel/

4 years ago by otterley

CDN edge servers maintain a warm pool of TCP/TLS connections to origin servers, which eliminates the need for a handshake and significantly improves performance.

4 years ago by CogitoCogito

I'm not an expert, but if the TLS handshake occurs closer (as in latency) to the client and the actual request/response payload is small (or if the CDN's network is better), then it seems like terminating in a CDN could certainly be better than not.

4 years ago by Cthulhu_

And CDN's offer flood / ddos protection as well.

4 years ago by dijit

I agree.

I've run some eCommerce sites before, we were PCI Tier 1, roughly 1% of web traffic: but nowhere _near_ the level of security Klarna should have; we had the CDN's activated only on subdomains... always.

Usually static.<site>.com or whatever..

This has the added benefit of not sending cookies to the CDN, because that can affect latency and is also a bit less secure because a static image/css/js does not need the cookie. (and bandwidth, before we were charged per byte (and that's egress now anyway in most cases, god I'm old.)).

If I had to venture a guess, they're probably scared of getting slammed with a DDoS so they "guard" everything behind the CDN, it's not about caching.

We got slammed with DDoS's too so we had to have systems in place to push BGP routes via verisign to "scrub" the traffic... we didn't even have that many "credits" and it cost millions...

4 years ago by notyourday

> We talked about this during the outage before but any API requests related to account information should not be going through a cacheable CDN.

That is simply not correct.

The entire reason why Fastly is so popular is because it lets you cache the API using cache-bust on write. Between the request collapsing, shielding, and a near immediate cache invalidation caching API allows one to accept enormous spikes of traffic that normally would bring down the origins.

4 years ago by johbjo

I suspect the background is the app is showing html-views rendered on the server, and they are sharing as much infrastructure as possible. To make it easier on the app team.

If they separated app/cdn/web with different SSL certificates, something like this wouldn't be possible even when developers eff up.

4 years ago by gherkinnn

Unrelated to the incident, but to Klarna.

What is Klarna _for_? What does Klarna do that others don't? They brand themselves as "We enable you to buy hip shit you don't have the money for [and will become irrelevant before you payed off your debt]". Almost literally [0]

Is it more of the same but dressed pretty or am I missing something?

That said, sometimes a common thing done up well will be successful. Looking at you, Slack.

0 - https://www.klarna.com/us/pay-in-4/

4 years ago by ullevaal

In Europe, strong customer authentication is now required when making credit card purchases https://en.wikipedia.org/wiki/Strong_customer_authentication

So Klarna delays having to authenticate with the bank which issued your credit card from the time of purchase, to the time of paying your Klarna "credit card bill".

They also delay sending your invoice until the item is actually shipped, and you then have a 14 day due date. This means returns and refunds have no money exchanged from any of your own accounts, so you don't have to worry about how long it takes for the refund to enter your credit card.

Edit: They also remember you address based on your email and name, across any vendors, so check out is really quick even if it is the first time you shop at a specific site. This is of course great for smaller vendors.

4 years ago by the_mitsuhiko

Klarna in Europe I think for most people I know is a "i made this mistake once" experience. I know nobody who picked the Klarna transaction type more than once.

The checkout is super quick only to be frustrated with the payment process later. And since it's Klarna that comes after you to collect payment it can eventually backfire on the business that you ordered from as well.

4 years ago by ullevaal

What's wrong with the payment process in your country? Where I am it's as easy as paying any invoice I get from a credit card company or I can by with debit card.

Also in my country, in Europe, Klarna is mentioned more and more in news articles about consumer debt and predatory lending, so I think their market share among people who keep credit card balances is increasing. They have stopped or kept pending unusually large purchases by me pending a credit check, but they could probably do a lot more to not get a reputation as lumped together with the most predatory consumer loan banks.

4 years ago by tut-urut-utut

Well, they are all in one payment processor, loan institute and debt collector with terrible record, with some questionable business practices.

4 years ago by __alexs

They are an elaborate pay day loans company.

4 years ago by samhw

Yeah, I would agree with that: it's payday loans with a friendly face. I once worked with someone who had previously been an exec at Klarna, and he had some 'interesting' stories about how they did credit decisioning. One of the examples that always stuck in my mind[0] was declining people with five or more vowels in their last name, because it was an effective proxy for West African people. (I remember that one because it also picks up my somewhat posh double-barrelled English surname, so I suppose I would have been collateral damage.)

[0] I should say I can't remember for sure if this was from his time at Klarna or somewhere else, but it was at least illustrative of the kind of people and the culture there.

4 years ago by rightbyte

Their main selling point is offering high interestrate credit to sites customers and paying the site the money directly, taking the risk.

4 years ago by geofft

This looks distressingly similar to https://blog.zulip.com/2021/03/20/zulip-cloud-security-incid... .

Klarna: Our app is configured specifically to avoid inadvertent caching through the use of cache control directives found in the HTTP standard (no-cache, no-store, must-revalidate). However, the CDN configuration can at times override the app configuration.

Zulip: Zulip separately marks its responses as non-cacheable, both to prevent private data from being sent to the wrong user, and also to prevent stale data from being sent even to the right user. Indicating that a response is not cacheable is the purpose of Zulip’s Cache-Control header: max-age=0 [...] no-cache [...] no-store [...] must-revalidate [...] private [...] CloudFront usually respects the Cache-Control and Expires headers, but with one critical exception when receiving multiple similar requests at roughly the same time

It looks like Klarna's main website uses CloudFront, so I bet this was the exact same thing.

4 years ago by tut-urut-utut

It's funny that I haven't heard about it before, although I live in Germany and Klarna is probably the biggest payment processor in Germany.

Luckily, I haven't used it. I never liked that one need to provide his bank login data to Klarna / Sofort to use it. How is that even legal?

EDIT: I haven't heard about Klarna outage, Klarna is well known.

4 years ago by julianlam

> I never liked that one need to provide his bank login data to Klarna / Sofort to use it. How is that even legal?

I am parroting other HN comments here, but the reason this is necessary is because banks do not provide a public (or private) API except between other banks. Therefore the only avenue for integration would be directly accessing the account.

4 years ago by Tainnor

Not really true in Germany where FinTS and its precursor HBCI have been supported by many banks since the 90s or so. Ironically, it's typically the hip start-up banks like N26 that don't support this standard.

4 years ago by ludamad

I wish this wasn't normal to me, but I've been fully conditioned to expect this from QuickBooksOnline

4 years ago by showsover

They do though. PDS2 requires all banks to have an accessible API.

4 years ago by samhw

This is different from banks actually _having_ an API. Many have stalled or successfully asked for exemptions, and even those who have gone ahead and, to some extent, complied, have still tried to make their APIs as difficult to use, feature-poor, and unreliable as possible. Source: worked at a bank which actually tried to do this well, and which also offered functionality which required us to integrate with other banks' APIs.

4 years ago by Tainnor

> How is that even legal?

It's worse than that. Several banks tried to enforce in their terms and conditions that sharing your bank login data would be a breach of contract or at least absolve the bank of any liability. This was deemed in violation of antitrust laws for some reason, so now banks have to actively allow this which is just brain-dead IMHO.

The only upside of this is that we now have PSD2 now anyway, so the damage that can be done by someone who has your login credentials is limited.

4 years ago by samhw

> This was deemed in violation of antitrust laws for some reason

My best guess at what that "some reason" may be is that it breaks shoddily built integrations like [Teller](https://teller.io/), which, instead of using schemes like Open Banking where available, simply rely on credential sharing.

4 years ago by mdaniel

For others wondering what PSD2 is: https://en.wikipedia.org/wiki/Payment_Services_Directive and there were also a lot of "what is it and how does it affect you" articles from name brand bank websites, too, but I figured start here

4 years ago by gmokki

In Finland it is illegal to give your bank login data to others, including scrapers. One couple both got sentenced when the other used the bank login data left on table to order something online. Fraud for the other and neglegence of leaving the login data easily accessible for the other.

4 years ago by randomlurking

I agree, haven’t used it (knowingly) and regularly order from other sites if klarna is the only payment option.

4 years ago by Xylakant

Are you certain you haven’t used it? Some shopping sites use them as payment processor for regular CC transactions without much visibility. You have to look at the T&C to even notice.

4 years ago by tut-urut-utut

I usually don't leave CC info on some random site, only to major and well known. If the site is unknown, it's either PayPal, "pay by invoice" or no order at all.

Either PayPal or CC issuer provide at least some buyers' protection, while given your bank credentials to Klarna is normally against ToC of your bank.

4 years ago by jokethrowaway

I try not to use PayPal in the hope they will become irrelevant as a business (I know, not holding my breath), mainly because their customer service and fees are terrible.

CCs are not a great mechanism and I wished we had better bank services and safer integrations to process payments, but that's what we have. I check my bank statement regularly and I know I can get money back if fraud happens. If I don't trust a website I used a prepaid with the exact amount on.

4 years ago by zibzab

> I usually don't leave CC info on some random site, only to major and well known.

That's the whole point of klarna (who also do CC processing so might have used them unknowingly): you don't need to trust Johnny's Little Online Store handling your CC, since the payment is handled by klarna.

4 years ago by undefined
[deleted]
4 years ago by Deathmax

This happened to Steam as well, where misconfigured caching leaked PII: https://store.steampowered.com/oldnews/19852

4 years ago by the-rc

It used to happen even with everything configured correctly, if your users connected over plain HTTP, from an ISP that forced their traffic to go through a transparent proxy with overaggressive caching. It's not fun to hear about users logged in with somebody else's account, but only in remote corners of the world instead of the more densely populated areas as you'd expect.

4 years ago by jokethrowaway

The same happened to the Italian government: https://moondo.info/en/Luciano-Vangone-suddenly-the-most-fam...

4 years ago by fuzzer37

> This configuration change had been written by an authorized engineer in one team, consulting but misinterpreting vendor documentation. The change was then reviewed by three other authorized engineers in three separate teams, according to normal processes. None of them spotted the mistake in the configuration update.

Is anyone else shocked by this? Are code reviews worthless? An engineer made a change reviewed by 3 other people. It seems like _someone_ would notice.

4 years ago by coldcode

Reviewing someone's code you had not seen before, involving a third party vendor you know nothing about, that only happens in production environment and only randomly fails and otherwise works—does not shock me at all. No process no matter how thorough can still fail. Code reviews in my prior experience solved nothing as people were expected to work full time and yet still review code continuously; unless you give engineers sufficient time to do both without overtime effort, one or the other will be affected, and usually its the reviews.

4 years ago by mdaniel

I had always seen code review process as partially a knowledge sharing exercise, so fine, maybe none of the 3 other engineers had previously seen those vendor docs, but part of the process is to retrace the steps and see if (with the benefit of hindsight) the reviewer would reach the same conclusion

> Code reviews in my prior experience solved nothing

I find that assertion suspicious, unless the culture was "push the approval button without even reading because ship,ship,ship"

4 years ago by peach

Once in a while a bunch of people will fail to notice something that might be cryptic if you don't know the details.

4 years ago by xbar

Impressive success at burying the lead.

Let me fix for it.

"Detailed incident report: No One At Klarna Understands That Disabling CDN Cache Protections for Personal Information is Bad"

4 years ago by brasetvik

Their initial post blamed the problem on “human error”, an anti-pattern in getting to the root cause of things.

This post attempts to blame their CDN vendor.

“All these reviewers couldn’t spot it.”

4 years ago by new_here

The blog post doesn't say much more than the headline. I'm curious about the specifics of what could of actually happened here.

In my limited experience working with CDNs wouldn't you just cache the responses of unique URLs and have some sort of cookie check at the edge before serving it.

So my own app would request something like /api/account?id=123 with my own id in there.

How would you end up getting other people's data in your app if your app only calls that unique URL?

4 years ago by samhw

It's pretty easy to imagine an API having an endpoint like GET /api/account/mine, which is implicitly parameterised by the user ID associated with that session. Or even a 'list' rather than a 'lookup' endpoint, like GET /api/messages, which fetched all private messages associated with the authenticated user – or whatever the equivalent private information would be, in Klarna's 'domain'.

Edit: If the other commenter is correct, then it's less bad than I imagined. Or rather it would at least only be triggered, seemingly, if someone deliberately and maliciously requested something that didn't 'belong' to them.

4 years ago by new_here

I see, so the it's likely that a generic URL pattern (like your example) was accidentally included in their caching rules.

I guess I didn't think of that originally because I thought if you wanted to cache some kind of response data like this, why on earth would you use a generic URL? But perhaps they probably didn't intend to cache this endpoint.

4 years ago by q3k

> How would you end up getting other people's data in your app if your app only calls that unique URL?

From what I understand what happened with this outage, the CDN would still cache /api/account?id=123, and someone with account ID 234 could access it by altering the URL to retrieve the cached version, if account 123 has used the app recently.

That's because a CDN has (usually) no concept of authorization/authentication and can't make decisions that /api/account?id=123 shouldn't be served to someone other than the owner of account 123.

It would be less catastrophic (at least from a PR point of view) because people wouldn't get immediately served others' accounts, but you'd be vulnerable to attack.

4 years ago by Ndymium

In fact users did get other users' data just by using the app normally, so it likely didn't happen the way you describe it.

4 years ago by q3k

Right, what probably happened is that an `/accounts/me` endpoint got cached.

But what GP seems to be asking about is: “Would having your app always encode a user ID into the endpoint have helped?”.

4 years ago by cuspycode

What I'm wondering is: why would you ever want a CDN configuration to override no-cache instructions from the backend? I assume there's a use case for this, but I can't figure out what it is. Can anyone explain?

Daily Digest

Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.