haketilo-hydrilla - An HTTPs proxy to facilitate replacing sites' js with user-supplied scripts

{#
SPDX-License-Identifier: GPL-3.0-or-later OR CC-BY-SA-4.0

Documentation page describing URL patterns understood by Haketilo.

This file is part of Hydrilla&Haketilo.

Copyright (C) 2022 Wojtek Kosior

Dual licensed under
* GNU General Public License v3.0 or later and
* Creative Commons Attribution Share Alike 4.0 International.

You can choose to use either of these licenses or both.


I, Wojtek Kosior, thereby promise not to sue for violation of this
file's licenses. Although I request that you do not make use of this
code in a proprietary work, I am not going to enforce this in court.
#}
{% extends "doc_base.html.jinja" %}

{% block title %} URL patterns {% endblock %}

{% block main %}
  {{ big_heading('Haketio URL patterns') }}

  {% call section() %}
    {% call paragraph() %}
      We want to be able to apply different rules and custom scripts for
      different websites. However, merely specifying "do this for all documents
      under <code>https://example.com</code>" is not enough. Single site's pages
      might differ strongly and require different custom scripts to be
      loaded. Always matching against a full URL like
      <code>https://example.com/something/somethingelse</code> is also not
      a good option. It doesn't allow us to properly handle a site that serves
      similar pages for multiple values substituted for
      <code>somethingelse</code>.
    {% endcall %}
  {% endcall %}

  {% call section() %}
    {{ medium_heading('Employed solution') }}

    {% call paragraph() %}
      Wildcards are being used to address the problem. Each payload and rule in
      Haketilo has a URL pattern that specifies to which internet pages it
      applies. A URL pattern can be as as simple as literal URL in which case it
      only matches itself. It can also contain wildcards in the form of one or
      more asterisks (<code>*</code>) that correspond to multiple possible
      strings occurring in that place.
    {% endcall %}

    {% call paragraph() %}
      Wildcards can appear in URL's domain and path that follows it. These 2
      types of wildcards are handled separately.
    {% endcall %}
  {% endcall %}

  {% call section() %}
    {{ small_heading('Domain wildcards') }}

    {% call paragraph() %}
      A domain wildcard takes the form of one, two or three asterisks occurring
      in place of a single domain name segment at the beginning
      (left). Depending on the number of asterisks, the meaning is as follows
    {% endcall %}

    {% call unordered_list() %}
      {% call list_entry() %}
        no asterisks (e.g. <code>example.com</code>) - match domain name exactly
        (e.g. <code>example.com</code>)
      {% endcall %}
      {% call list_entry() %}
        one asterisk (e.g. <code>*.example.com</code>) - match all domains
        resulting from substituting <code>*</code> with a
        <span class="bold">single</span> segment (e.g.
        <code>banana.example.com</code> or <code>pineapple.example.com</code>
        but <span class="bold">not</span> <code>pineapple.pen.example.com</code>
        nor <code>example.com</code>)
      {% endcall %}
      {% call list_entry() %}
        two asterisks (e.g. <code>**.example.com</code>) - match all domains
        resulting from substituting <code>**</code> with
        <span class="bold">two or more</span> segments (e.g.
        <code>monad.breakfast.example.com</code> or
        <code>pure.monad.breakfast.example.com</code> but
        <span class="bold">not</span> <code>cabalhell.example.com</code> nor
        <code>example.com</code>)
      {% endcall %}
      {% call list_entry() %}
        three asterisks (e.g. <code>***.example.com</code>) - match all domains
        resulting from substituting <code>***</code> with
        <span class="bold">zero or more</span> segments (e.g.
        <code>hello.parkmeter.example.com</code> or
        <code>iliketrains.example.com</code> or <code>example.com</code>)
      {% endcall %}
    {% endcall %}
  {% endcall %}

  {% call section() %}
    {{ small_heading('Path wildcards') }}

    {% call paragraph() %}
      A path wildcard takes the form of one, two or three asterisks occurring in
      place of a single path segment at the end of path (right). Depending on
      the number of asterisks, the meaning is as follows
    {% endcall %}

    {% call unordered_list() %}
      {% call list_entry() %}
        no asterisks (e.g. <code>/joke/clowns</code>) - match path exactly (e.g.
        <code>/joke/clowns</code>)
      {% endcall %}
      {% call list_entry() %}
        one asterisk (e.g. <code>/itscalled/*</code>) - match all paths
        resulting from substituting <code>*</code> with a
        <span class="bold">single</span> segment (e.g.
        <code>/itscalled/gnulinux</code> or <code>/itscalled/glamp</code> but
        <span class="bold">not</span> <code>/itscalled/</code> nor
        <code>/itscalled/gnu/linux</code>)
      {% endcall %}
      {% call list_entry() %}
        two asterisks (e.g. <code>/another/**</code>) - match all paths
        resulting from substituting <code>**</code> with
        <span class="bold">two or more</span> segments (e.g.
        <code>/another/nsa/backdoor</code> or
        <code>/another/best/programming/language</code> but
        <span class="bold">not</span> <code>/another/apibreak</code> nor
        <code>/another</code>)
      {% endcall %}
      {% call list_entry() %}
        three asterisks (e.g. <code>/mail/dmarc/***</code>) - match all paths
        resulting from substituting <code>***</code> with
        <span class="bold">zero or more</span> segments (e.g.
        <code>/mail/dmarc/spf</code>, <code>/mail/dmarc</code> or
        <code>/mail/dmarc/dkim/failure</code> but
        <span class="bold">not</span> <code>/mail/</code>)
      {% endcall %}
    {% endcall %}

    {% call paragraph() %}
      If pattern ends <span class="bold">without</span> a trailing slash, it
      mathes paths with any number of trailing slashes, including zero. If
      pattern ends <span class="bold">with</span> a trailing slash, it only
      mathes paths with one or more trailing slashes. For example,
      <code>/itscalled/*</code> matches <code>/itscalled/gnulinux</code>,
      <code>/itscalled/gnulinux/</code> and <code>/itscalled/gnulinux//</code>
      while <code>/itscalled/*/</code> only matches
      <code>/itscalled/gnulinux/</code> and <code>/itscalled/gnulinux//</code>
      out of those three.
    {% endcall %}

    {% call paragraph() %}
      If two patterns only differ by the presence of a trailing slash,
      pattern <span class="bold">with</span> a trailing slash is considered
      <span class="bold">more specific</span>.
    {% endcall %}

    {% call paragraph() %}
      Additionally, any path with literal trailing asterisks is matched by
      itself, even if such pattern would otherwise be treated as wildcard
      (e.g. <code>/gobacktoxul/**</code> matches <code>/gobacktoxul/**</code>).
      This is likely to change in the future and would best not be relied upon.
      Appending three additional asterisks to path pattern to represent literal
      asterisks is being considered.
    {% endcall %}
  {% endcall %}

  {% call section() %}
    {{ small_heading('URL scheme wildcard') }}

    {% call paragraph() %}
      <code>http://</code> and <code>https://</code> shemes in the URL are
      matched exactly. However, starting with Haketilo 3.0, it is also possible
      for scheme pseudo-wildcard of <code>http*://</code> to be used. Use of URL
      pattern with this scheme is equivalent to the use of 2 separate patterns
      starting with <code>http://</code> and <code>https://</code>,
      respectively. For example, pattern <code>http*://example.com</code> shall
      match both <code>https://example.com</code> and
      <code>http://example.com</code>.
    {% endcall %}

    {% call paragraph() %}
      <code>http*://</code> may be considered not to be a true wildcard but
      rather an alias for either of the other 2 values. As of Haketilo 3.0, the
      speicificity of a URL pattern starting with <code>http*://</code> is
      considered to be the same as that of the corresponding URL pattern
      starting with <code>http://</code> or <code>https://</code>. In case of a
      conflict, the order of precedence of such patterns is unspecified. This
      behavior is likely to change in the future versions of Haketilo.
    {% endcall %}
  {% endcall %}

  {% call section() %}
    {{ small_heading('Wildcard pattern priorities and querying') }}

    {% call paragraph() %}
      In case multiple patterns match some URL, the more specific one is
      preferred. Specificity is considered as follows
    {% endcall %}

    {% call unordered_list() %}
      {% call list_entry() %}
        If patterns only differ in the final path segment, the one with least
        wildcard asterisks in that segment if preferred.
      {% endcall %}
      {% call list_entry() %}
        If patterns, besides the above, only differ in path length, one with
        longer path is preferred. Neither final wildcard segment nor trailing
        dashes account for path length.
      {% endcall %}
      {% call list_entry() %}
        If patterns, besides the above, only differ in the initial domain
        segment, one with least wildcard asterisks in that segment is preferred.
      {% endcall %}
      {% call list_entry() %}
        If patterns differ in domain length, one with longer domain is
        preferred. Initial wildcard segment does not account for domain length.
      {% endcall %}
    {% endcall %}

    {% call paragraph() %}
      As an example, consider the URL
      <code>http://settings.query.example.com/google/tries/destroy/adblockers//</code>.
      Patterns matching it are, in the following order
    {% endcall %}

    {% call verbatim() %}
http://settings.query.example.com/google/tries/destroy/adblockers/
http://settings.query.example.com/google/tries/destroy/adblockers
http://settings.query.example.com/google/tries/destroy/adblockers/***/
http://settings.query.example.com/google/tries/destroy/adblockers/***
http://settings.query.example.com/google/tries/destroy/*/
http://settings.query.example.com/google/tries/destroy/*
http://settings.query.example.com/google/tries/destroy/***/
http://settings.query.example.com/google/tries/destroy/***
http://settings.query.example.com/google/tries/**/
http://settings.query.example.com/google/tries/**
http://settings.query.example.com/google/tries/***/
http://settings.query.example.com/google/tries/***
http://settings.query.example.com/google/**/
http://settings.query.example.com/google/**
http://settings.query.example.com/google/***/
http://settings.query.example.com/google/***
http://settings.query.example.com/**/
http://settings.query.example.com/**
http://settings.query.example.com/***/
http://settings.query.example.com/***
http://***.settings.query.example.com/google/tries/destroy/adblockers/
http://***.settings.query.example.com/google/tries/destroy/adblockers
http://***.settings.query.example.com/google/tries/destroy/adblockers/***/
http://***.settings.query.example.com/google/tries/destroy/adblockers/***
http://***.settings.query.example.com/google/tries/destroy/*/
http://***.settings.query.example.com/google/tries/destroy/*
http://***.settings.query.example.com/google/tries/destroy/***/
http://***.settings.query.example.com/google/tries/destroy/***
http://***.settings.query.example.com/google/tries/**/
http://***.settings.query.example.com/google/tries/**
http://***.settings.query.example.com/google/tries/***/
http://***.settings.query.example.com/google/tries/***
http://***.settings.query.example.com/google/**/
http://***.settings.query.example.com/google/**
http://***.settings.query.example.com/google/***/
http://***.settings.query.example.com/google/***
http://***.settings.query.example.com/**/
http://***.settings.query.example.com/**
http://***.settings.query.example.com/***/
http://***.settings.query.example.com/***
http://*.query.example.com/google/tries/destroy/adblockers/
http://*.query.example.com/google/tries/destroy/adblockers
http://*.query.example.com/google/tries/destroy/adblockers/***/
http://*.query.example.com/google/tries/destroy/adblockers/***
http://*.query.example.com/google/tries/destroy/*/
http://*.query.example.com/google/tries/destroy/*
http://*.query.example.com/google/tries/destroy/***/
http://*.query.example.com/google/tries/destroy/***
http://*.query.example.com/google/tries/**/
http://*.query.example.com/google/tries/**
http://*.query.example.com/google/tries/***/
http://*.query.example.com/google/tries/***
http://*.query.example.com/google/**/
http://*.query.example.com/google/**
http://*.query.example.com/google/***/
http://*.query.example.com/google/***
http://*.query.example.com/**/
http://*.query.example.com/**
http://*.query.example.com/***/
http://*.query.example.com/***
http://***.query.example.com/google/tries/destroy/adblockers/
http://***.query.example.com/google/tries/destroy/adblockers
http://***.query.example.com/google/tries/destroy/adblockers/***/
http://***.query.example.com/google/tries/destroy/adblockers/***
http://***.query.example.com/google/tries/destroy/*/
http://***.query.example.com/google/tries/destroy/*
http://***.query.example.com/google/tries/destroy/***/
http://***.query.example.com/google/tries/destroy/***
http://***.query.example.com/google/tries/**/
http://***.query.example.com/google/tries/**
http://***.query.example.com/google/tries/***/
http://***.query.example.com/google/tries/***
http://***.query.example.com/google/**/
http://***.query.example.com/google/**
http://***.query.example.com/google/***/
http://***.query.example.com/google/***
http://***.query.example.com/**/
http://***.query.example.com/**
http://***.query.example.com/***/
http://***.query.example.com/***
http://**.example.com/google/tries/destroy/adblockers/
http://**.example.com/google/tries/destroy/adblockers
http://**.example.com/google/tries/destroy/adblockers/***/
http://**.example.com/google/tries/destroy/adblockers/***
http://**.example.com/google/tries/destroy/*/
http://**.example.com/google/tries/destroy/*
http://**.example.com/google/tries/destroy/***/
http://**.example.com/google/tries/destroy/***
http://**.example.com/google/tries/**/
http://**.example.com/google/tries/**
http://**.example.com/google/tries/***/
http://**.example.com/google/tries/***
http://**.example.com/google/**/
http://**.example.com/google/**
http://**.example.com/google/***/
http://**.example.com/google/***
http://**.example.com/**/
http://**.example.com/**
http://**.example.com/***/
http://**.example.com/***
http://***.example.com/google/tries/destroy/adblockers/
http://***.example.com/google/tries/destroy/adblockers
http://***.example.com/google/tries/destroy/adblockers/***/
http://***.example.com/google/tries/destroy/adblockers/***
http://***.example.com/google/tries/destroy/*/
http://***.example.com/google/tries/destroy/*
http://***.example.com/google/tries/destroy/***/
http://***.example.com/google/tries/destroy/***
http://***.example.com/google/tries/**/
http://***.example.com/google/tries/**
http://***.example.com/google/tries/***/
http://***.example.com/google/tries/***
http://***.example.com/google/**/
http://***.example.com/google/**
http://***.example.com/google/***/
http://***.example.com/google/***
http://***.example.com/**/
http://***.example.com/**
http://***.example.com/***/
http://***.example.com/***
    {% endcall %}

    {% call paragraph() %}
      Variants of those patterns starting with <code>http*://</code> would of
      course match as well. They have been omitted for simplicity.
    {% endcall %}

    {% call paragraph() %}
      For a simpler URL like <code>https://example.com</code> the patterns would
      be
    {% endcall %}

    {% call verbatim() %}
https://example.com
https://example.com/***
https://***.example.com
https://***.example.com/***
    {% endcall %}

    {% call paragraph() %}
      Variants of those patterns with a trailing dash added
      would <span class="bold">not</span> match the URL. Also, the pattern
      variants starting with <code>http*://</code> have been once again omitted.
    {% endcall %}
  {% endcall %}

  {% call section() %}
    {{ small_heading('Limits') }}

    {% call paragraph() %}
      In order to prevent some easy-to-conduct DoS attacks, older versions of
      Haketilo and Hydrilla limited the lengths of domain and path parts of
      processed URLs. This is no longer the case.
    {% endcall %}
  {% endcall %}

  {% call section() %}
    {{ medium_heading('Alternative solution idea: mimicking web server mechanics') }}

    {% call paragraph() %}
      While wildcard patterns as presented give a lot of flexibility, they are
      not the only viable approach to specifying what URLs to apply
      rules/payloads to. In fact, wildcards are different from how the server
      side of a typical website decides what to return for a given URL request.
    {% endcall %}

    {% call paragraph() %}
      In a typical scenario, an HTTP server like Apache reads configuration
      files provided by its administrator and uses various (virtual host,
      redirect, request rewrite, CGI, etc.) instructions to decide how to handle
      given URL. Perhps using a scheme that mimics the configuration options
      typically used with web servers would give more efficiency in specifying
      what page settings to apply when.
    {% endcall %}

    {% call paragraph() %}
      This approach may be considered in the future.
    {% endcall %}
  {% endcall %}
{% endblock main %}