Incorrect protocol when deployed behind AWS ALB

Hello,

We have a Bloomreach deployment on AWS that looks like the following:

External ALB (80/443) → nginx (80/443) → Internal ALB (8080) → Bloomreach HST (8080)

The architecture includes an external ALB (application load balancer) which reverse proxies back to nginx instances. These nginx instances then forward (most) requests back to an internal ALB which in turn proxies requests back to Bloomreach HST instances.

Right now, the Bloomreach instance sees the incorrect value of X-Forwarded-Proto. This is because the Internal ALB overwrites the value of the header to ‘http’, because the internal ALB is configured with HTTP listeners only. It doesn’t appear to be possible to preserve the value of the header using ALBs.

One way to fix this might be to write a servlet filter early in the request cycle to fix the value of the X-Forwarded-Proto before handing off to the Hippo filter chain. We’d have to do that for each instance of Bloomreach.

Another possible fix would be to replace the internal ALB with an NLB (network load balancer), thus stopping it from trying to do anything with HTTP headers. However, using an ALB provides some useful monitoring, so that’s not without consequence.

I wondered if other folk have run into the same problem, and how they’ve solved it?

Also, is there another header I can add to the request using nginx that will take precedence over the overwritten X-Forwarded-Proto header?

Thanks
Martin

It is unfortunate that ALB doesn’t append the protocol to make it up to X-Forwarded-Proto: https,http (some reverse proxy servers do this).
Anyway, HST relies on HstRequestUtils#getFarthestRequestScheme() [1].
Not sure yet what could be done there…
Any ideas or PR?

Cheers,

Woonsan

[1] brxm/HstRequestUtils.java at brxm-14.5.0-1 · bloomreach/brxm · GitHub

Hi Woonsan,

The X-Forwarded-Proto header shouldn’t be comma-separated in the same way as the X-Forwarded-For header, at least according to the MDN documentation. It looks like it’s just supposed to have a single protocol value, so that header doesn’t seem quite right for what we need.

One option would be to implement a proprietary header, say X-Bloomreach-Proto or X-HST-Proto, that overrides the standard header. Nginx could set that header, and I assume it would be forwarded on without modification by pretty much any load balancer.

Another option might be to implement support for the Forwarded header. Our nginx proxy could add that to the request.

If ALBs ever added support for the Forwarded header they would presumably do this by appending either a comma-separated value, or adding a new header (with the same name). I’m assuming ordering is the same as for the X-Forwarded-For header. That shouldn’t then break anything, because Bloomreach would need to look at the first comma-separated value of the first Forwarded header to determine the correct protocol.

What do you think?

Thanks
Martin

Hi Martin,

The first option, possibly with X-BR-Forwarded-Proto, seems simpler.
Would you mind creating a JIRA ticket at https://issues.onehippo.com/browse/CMS?

Thanks,

Woonsan

Thanks, Woonsan.

I have created CMS-14725.

I’ve also suggested another approach: adding a configuration value that identifies the name of the header to use, which could default to X-Forwarded-Proto.

This would be consistent with support for custom names for the X-Forwarded-For header described in Custom Resolution of Client’s Originating IP Address. The logic for that is implemented in the same class, so it may be possible to reuse some of it.

Martin