mark/surplus
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

v2.1.0 (#21)

Browse source
This commit is contained in:
Mark Joshwel 2023-09-07 02:49:20 +08:00 committed by GitHub
parent bf26afb95a 1ac2e1015b
commit 7a85737fd4
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
11 changed files with 1700 additions and 354 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
.github/workflows/checks.yml vendored
View file

@ -4,6 +4,7 @@ on:
workflow_dispatch:
push:
paths:
- '*.py'
- '**.py'
jobs:
@ -24,13 +25,13 @@ jobs:
run: devbox run poetry build
- name: analyse with mypy
run: devbox run poetry run mypy **/*.py
run: devbox run poetry run mypy .
- name: check for black formatting compliance
run: devbox run poetry run "black --check **/*.py"
run: devbox run poetry run "black --check ."
- name: analyse isort compliance
run: devbox run poetry run "isort --check **/*.py"
run: devbox run poetry run "isort --check *.py **/*.py"
test:
runs-on: ubuntu-latest

60
.github/workflows/publish-slsa3-auto.yml vendored Normal file
View file

@ -0,0 +1,60 @@
name: automated tagged release with slsa 3 compliance
on:
push:
tags:
- '*'
jobs:
build:
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
permissions:
contents: write
outputs:
hashes: ${{ steps.hash.outputs.hashes }}
steps:
- name: checkout
uses: actions/checkout@v3
- name: install devbox
uses: jetpack-io/devbox-install-action@v0.3.0
- name: install dependencies
run: devbox run poetry install
- name: run releaser.py
run: devbox run python releaser.py
- name: build project
id: build
run: devbox run poetry build
- name: duplicate non-versioned wheel
run: cp dist/surplus-*.whl dist/surplus-latest-py3-none-any.whl
- name: generate provenance subjects
id: hash
run: |
cd dist
HASHES=$(sha256sum * | base64 -w0)
echo "hashes=$HASHES" >> "$GITHUB_OUTPUT"
- name: release
uses: softprops/action-gh-release@v0.1.15
if: startsWith(github.ref, 'refs/tags/')
with:
files: |
dist/*.whl
provenance:
needs: [build]
permissions:
actions: read
id-token: write
contents: write
uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v1.6.0
with:
base64-subjects: "${{ needs.build.outputs.hashes }}"
upload-assets: true

11
.github/workflows/publish-slsa3.yml → .github/workflows/publish-slsa3-manual.yml vendored
View file

@ -1,9 +1,7 @@
name: release with slsa 3 compliance
name: manual release with slsa 3 compliance
on:
push:
tags:
- '*'
workflow_dispatch:
jobs:
build:
@ -23,7 +21,10 @@ jobs:
- name: install dependencies
run: devbox run poetry install
- name: install dependencies
- name: run releaser.py
run: devbox run python releaser.py
- name: build project
id: build
run: devbox run poetry build

484
README.md
View file

@ -4,6 +4,13 @@ surplus is a Python script to convert
[Google Maps Plus Codes](https://maps.google.com/pluscodes/)
to iOS Shortcuts-like shareable text.
> [!NOTE]
> you are on the `future` branch.
> this branch contains the latest development changes to surplus, but may be unstable.
>
> checkout the [`main`](https://github.com/markjoshwel/surplus/tree/main) branch for the
> a stable version of the repository.
- [installation](#installation)
- [usage](#usage)
- [command-line usage](#command-line-usage)
@ -15,11 +22,12 @@ to iOS Shortcuts-like shareable text.
- [what counts as "incorrect"](#what-counts-as-incorrect)
- [output technical details](#the-technical-details-of-surpluss-output)
- [api reference](#api-reference)
- [details on the fingerprinted user agent](#details-on-the-fingerprinted-user-agent)
- [licence](#licence)
```text
$ surplus 9R3J+R9 Singapore
surplus version 2.0.1
surplus version 2.1.0
Thomson Plaza
301 Upper Thomson Road
Sin Ming, Bishan
@ -55,7 +63,8 @@ see [licence](#licence) for licensing information.
### command-line usage
```text
usage: surplus [-h] [-d] [-v] [-c {pluscode,localcode,latlong,string}]
usage: surplus [-h] [-d] [-v] [-c {pluscode,localcode,latlong,sharetext}]
[-u USER_AGENT]
[query ...]
Google Maps Plus Code to iOS Shortcuts-like shareable text
@ -63,25 +72,27 @@ Google Maps Plus Code to iOS Shortcuts-like shareable text
positional arguments:
query full-length Plus Code (6PH58QMF+FX), shortened
Plus Code/'local code' (8QMF+FX Singapore),
latlong (1.3336875, 103.7749375), or string
query (e.g., 'Wisma Atria')
latlong (1.3336875, 103.7749375), string query
(e.g., 'Wisma Atria'), or '-' to read from stdin
options:
-h, --help show this help message and exit
-d, --debug prints lat, long and reverser response dict to
stderr
-v, --version prints version information to stderr and exits
-c {pluscode,localcode,latlong,sharetext},
--convert-to {pluscode,localcode,latlong,sharetext}
-c {pluscode,localcode,latlong,sharetext}, --convert-to {pluscode,localcode,latlong,sharetext}
converts query a specific output type, defaults
to 'sharetext'
-u USER_AGENT, --user-agent USER_AGENT
user agent string to use for geocoding service,
defaults to fingerprinted user agent string
```
### example api usage
here are a few examples to get you quickly started using surplus in your own program:
1. let surplus do the heavy lifiting
1. let surplus do the heavy lifting
```python
>>> from surplus import surplus, Behaviour
@ -90,7 +101,7 @@ here are a few examples to get you quickly started using surplus in your own pro
'Ngee Ann Polytechnic\n535 Clementi Road\nBukit Timah\n599489\nNorthwest, Singapore'
```
2. handle queries seperately
2. handle queries separately
```python
>>> import surplus
@ -115,7 +126,7 @@ here are a few examples to get you quickly started using surplus in your own pro
notes:
- you can change what surplus does by passing in a custom [`Behaviour`](#class-behaviour)
- you can change what surplus does when passing in a custom [`Behaviour`](#class-behaviour)
object
- most surplus functions return a [`Result`](#class-result) object. while you can
@ -167,7 +178,7 @@ the command to create an empty commit is `git commit --allow-empty`
### reporting incorrect output
> [!NOTE]
> this section is independent from the rest of the contributing section.
> this section is independent of the rest of the contributing section.
different output from the iOS Shortcuts app is expected, however incorrect output is not.
@ -181,13 +192,13 @@ and do the following:
function, which by default is OpenStreetMap Nominatim.
(_don't know what the above means? then you are using the default reverser._)
also look at the [what counts as "incorrect"](#what-counts-as-incorrect) section
also look at the ['what counts as "incorrect"'](#what-counts-as-incorrect) section
before moving on.
2. include the erroneous query.
(_the Plus Code/local code/latlong coord/query string you passed into surplus_)
(_the Plus Code/local code/latlong coordinate/query string you passed into surplus_)
3. include output from the teminal with the
3. include output from the terminal with the
[`--debug` flag](#command-line-usage) passed to the surplus CLI or with
`debug=True` set in function calls.
@ -234,8 +245,9 @@ other examples that _should not_ be reported are:
- name of place is incorrect/different
this may be due to incorrect data from the geolocator function, which is OpenStreetMap Nominatim by default.
in the case of Nominatim, it means that the data on OpenStreetMap is incorrect.
this may be due to incorrect data from the geocoder function, which is OpenStreetMap
Nominatim by default. in the case of Nominatim, it means that the data on OpenStreetMap
is incorrect.
(_if so, then consider updating OpenStreetMap to help not just you, but other surplus
and OpenStreetMap users!_)
@ -255,12 +267,12 @@ of incorrect outputs.
```text
$ s+ --debug 8QJF+RP Singapore
surplus version 2.0.1, debug mode
surplus version 2.1.0, debug mode (latest@future, Tue 05 Sep 2023 23:38:59 +0800)
debug: parse_query: behaviour.query=['8QJF+RP', 'Singapore']
debug: _match_plus_code: portion_plus_code='8QJF+RP', portion_locality='Singapore'
debug: cli: query=Result(value=LocalCodeQuery(code='8QJF+RP', locality='Singapore'), error=None)
debug: cli: latlong.get()=Latlong(latitude=1.3320625, longitude=103.7743125)
debug: cli: location={'amenity': 'Ngee Ann Polytechnic', 'house_number': '535', 'road': 'Clementi Road', 'suburb': 'Bukit Timah', 'city': 'Singapore', 'county': 'Northwest', 'ISO3166-2-lvl6': 'SG-03', 'postcode': '599489', 'country': 'Singapore', 'country_code': 'sg', 'raw': "{...}", 'latitude': '1.33318835', 'longitude': '103.77461234638255'}
debug: latlong_result.get()=Latlong(latitude=1.3320625, longitude=103.7743125)
debug: location={...}
debug: _generate_text: seen_names=['Ngee Ann Polytechnic', 'Clementi Road']
debug: _generate_text_line: [True] -> True -------- 'Ngee Ann Polytechnic'
debug: _generate_text_line: [True] -> True -------- '535'
@ -286,32 +298,43 @@ Northwest, Singapore
variables
- **variable `behaviour.query`**
- **variables `behaviour.query`, `split_query` and `original_query`**
the original query string or a list of strings from space-splitting the original query
(_`split_query` and `original_query` are only shown if query is a latlong coordinate
or query string_)
`behaviour.query` is the original query string or a list of strings from space-splitting the original query
string passed to [`parse_query()`](#def-parse_query) for parsing
`split_query` is the original query string split by spaces
`original_query` is a single non-split string
```text
$ s+ 77Q4+7X Austin, Texas, USA
--------------------------
$ s+ Temasek Polytechnic
-------------------
query
behaviour.query -> ['77Q4+7X', 'Austin', 'Texas', 'USA']
behaviour.query -> ['Temasek', 'Polytechnic']
split_query -> ['Temasek', 'Polytechnic']
original_query -> 'Temasek Polytechnic'
```
```text
>>> surplus("77Q4+7X Austin, Texas, USA", surplus.Behaviour())
behaviour.query -> '77Q4+7X Austin, Texas, USA'
split_query -> ['77Q4+7X', 'Austin,', 'Texas,', 'USA']
original_query -> '77Q4+7X Austin, Texas, USA'
```
- **variables `portion_plus_code` and `portion_locality`**
(_only shown if the query is a local code, not shown on full-length plus codes,
(_only shown if the query is a local code, not shown on full-length Plus Codes,
latlong coordinates or string queries_)
represents the plus code and locality portions of a
[shortened plus code](https://en.wikipedia.org/wiki/Open_Location_Code#Common_usage_and_shortening)
represents the Plus Code and locality portions of a
[shortened Plus Code](https://en.wikipedia.org/wiki/Open_Location_Code#Common_usage_and_shortening)
(_referred to as a "local code" in the codebase_) respectively
- **variable `query`**
@ -321,23 +344,23 @@ variables
this variable is displayed to show what query type [`parse_query()`](#def-parse_query) has
recognised, and if there were any errors during query parsing
- **expression `latlong.get()=`**
- **expression `latlong_result.get()=`**
(_only shown if the query is a plus code_)
(_only shown if the query is a Plus Code_)
the latitude longitude coordinates derived from the plus code
the latitude longitude coordinates derived from the Plus Code
- **variable `location`**
the response dictionary from the reverser function passed to
[`surplus()`](#def-surplus)
for more information on the reverser function, see [`Behaviour`](#class-behaviour) and
[`default_reverser`](#def-default_reverser)
for more information on the reverser function, see
[`SurplusReverserProtocol`](#surplusreverserprotocol)
- **variable `seen_names`**
a list of unique important names found in certain nominatim keys used in final output
a list of unique important names found in certain Nominatim keys used in final output
lines 0-3
- **`_generate_text_line` seen name checks**
@ -516,7 +539,13 @@ line breakdown of shareable text output, accompanied by their Nominatim keys:
- [types](#types)
- [`Query`](#query)
- [`ResultType`](#resulttype)
- [`SurplusGeocoderProtocol`](#surplusgeocoderprotocol)
- [`SurplusReverserProtocol`](#surplusreverserprotocol)
- [`class Behaviour`](#class-behaviour)
- [`class SurplusDefaultGeocoding`](#class-surplusdefaultgeocoding)
- [`SurplusDefaultGeocoding.update_geocoding_functions()`](#surplusdefaultgeocodingupdate_geocoding_functions)
- [`SurplusDefaultGeocoding.geocoder()`](#surplusdefaultgeocodinggeocoder)
- [`SurplusDefaultGeocoding.reverser()`](#surplusdefaultgeocodingreverser)
- [`class ConversionResultTypeEnum`](#class-conversionresulttypeenum)
- [`class Result`](#class-result)
- [`Result.__bool__()`](#result__bool__)
@ -539,8 +568,8 @@ line breakdown of shareable text output, accompanied by their Nominatim keys:
- [`StringQuery.__str__()`](#stringquery__str__)
- [`def surplus()`](#def-surplus)
- [`def parse_query()`](#def-parse_query)
- [`def default_geocoder()`](#def-default_geocoder)
- [`def default_reverser()`](#def-default_reverser)
- [`def generate_fingerprinted_user_agent`](#def-generate_fingerprinted_user_agent)
- [details on the fingerprinted user agent](#details-on-the-fingerprinted-user-agent)
### constants
@ -549,22 +578,55 @@ line breakdown of shareable text output, accompanied by their Nominatim keys:
a tuple of integers representing the version of surplus, in the format
`[major, minor, patch]`
- `SHAREABLE_TEXT_LINE_0_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_1_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_2_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_3_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_4_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_5_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_6_KEYS: tuple[str, ...]`
- `VERSION_SUFFIX: typing.Final[str]`
`BUILD_BRANCH: typing.Final[str]`
`BUILD_COMMIT: typing.Final[str]`
`BUILD_DATETIME: typing.Final[datetime]`
a tuple of strings containing nominatim keys used in shareable text line 0-6
string and a [datetime.datetime](https://docs.python.org/3/library/datetime.html) object
containing version and build information, set by [releaser.py](releaser.py)
- `SHAREABLE_TEXT_NAMES: tuple[str, ...]`
- `CONNECTION_MAX_RETRIES: int = 9`
`CONNECTION_WAIT_SECONDS: int = 10`
a tuple of strings containing nominatim keys used in shareable text line 0-2 and
defines if and how many times to retry a connection, alongside how many seconds to wait
in between tries, for Nominatim
> [!NOTE]
> this constant only affects the default surplus Nominatim geocoding functions. custom
> functions do not read from this, unless deliberately programmed to do so
- `SHAREABLE_TEXT_LINE_0_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_1_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_2_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_3_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_4_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_5_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_6_KEYS: typing.Final[tuple[str, ...]]`
a tuple of strings containing Nominatim keys used in shareable text line 0-6
- `SHAREABLE_TEXT_NAMES: typing.Final[tuple[str, ...]]`
a tuple of strings containing Nominatim keys used in shareable text line 0-2 and
special keys in line 3
- `EMPTY_LATLONG: Latlong`
- `SHAREABLE_TEXT_LOCALITY: dict[str, tuple[str, ...]]`
a dictionary of iso3166-2 country-portion strings with a tuples of strings as their
values
used when generating the locality portions of shortened Plus Codes/local codes
```python
{
"default": (...),
"SG": (...,),
...
}
```
- `EMPTY_LATLONG: typing.Final[Latlong]`
a constant for an empty latlong coordinate, with latitude and longitude set to 0.0
### exception classes
@ -602,6 +664,91 @@ ResultType = TypeVar("ResultType")
[generic type](https://docs.python.org/3/library/typing.html#generics) used by
[`Result`](#class-result)
#### `SurplusGeocoderProtocol`
[typing_extensions.Protocol](https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols)
class for documentation and static type checking of surplus geocoder functions
- **signature and conforming function signature**
```python
class SurplusGeocoderProtocol(Protocol):
def __call__(self, place: str) -> Latlong:
...
```
functions that conform to this protocol should have the following signature:
```python
def example(place: str) -> Latlong: ...
```
- **information on conforming functions**
function takes in a location name as a string, and returns a [Latlong](#class-latlong).
**function MUST supply a `bounding_box` attribute to the to-be-returned
[Latlong](#class-latlong).** the bounding box is used when surplus shortens Plus Codes.
function can and should be at minimum
[`functools.lru_cache()`-wrapped](https://docs.python.org/3/library/functools.html#functools.lru_cache)
if the geocoding service asks for caching
exceptions are handled by the caller
#### `SurplusReverserProtocol`
[typing_extensions.Protocol](https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols)
class for documentation and static type checking of surplus reverser functions
- **signature and conforming function signature**
```python
class SurplusReverserProtocol(Protocol):
def __call__(self, latlong: Latlong, level: int = 18) -> dict[str, Any]:
...
```
functions that conform to this protocol should have the following signature:
```python
def example(self, latlong: Latlong, level: int = 18) -> dict[str, Any]: ...
```
- **information on conforming functions**
function takes in a [Latlong](#class-latlong) object and return a dictionary with [`SHAREABLE_TEXT_LINE_*_KEYS`](#constants) keys at the dictionaries' top-level.
keys are used to access address information.
function should also take in an int representing the level of detail for the returned
address, 0-18 (country-level to building), inclusive. should default to 18.
keys for latitude, longitude and an iso3166-2 (or closest equivalent) should also be
included at the dictionaries top level as the keys `latitude`, `longitude` and
`ISO3166-2` (non-case sensitive, or at least something starting with `ISO3166`)
respectively.
```python
{
'ISO3166-2-lvl6': 'SG-03',
'amenity': 'Ngee Ann Polytechnic',
...
'country': 'Singapore',
'latitude': 1.33318835,
'longitude': 103.77461234638255,
'postcode': '599489',
'raw': {...},
}
```
function can and should be at minimum
[`functools.lru_cache()`-wrapped](https://docs.python.org/3/library/functools.html#functools.lru_cache)
if the geocoding service asks for caching
see the [playground notebook](/playground.ipynb) in repository root for detailed
sample output
exceptions are handled by the caller
### `class Behaviour`
[`typing.NamedTuple`](https://docs.python.org/3/library/typing.html#typing.NamedTuple)
@ -613,15 +760,13 @@ attributes
original user-passed query string or a list of strings from splitting user-passed query
string by spaces
- `geocoder: typing.Callable[[str], Latlong] = default_geocoder`
name string to location function, must take in a string and return a
[`Latlong`](#class-latlong), exceptions are handled by the caller
- `geocoder: SurplusGeocoderProtocol = default_geocoding.geocoder`
name string to location function, see
[`SurplusGeocoderProtocol`](#surplusgeocoderprotocol) for more information
- `reverser: Callable[[Latlong], dict[str, Any]] = default_reverser`
[`Latlong`](#class-latlong) object to dictionary function, must take in a string and return a
dict. keys found in SHAREABLE_TEXT_LINE_*_KEYS used to access address details are placed
top-level in the dict, exceptions are handled by the caller.
see the [playground notebook](playground.ipynb) for example output
- `reverser: SurplusReverserProtocol = default_geocoding.reverser`
Latlong object to address information dictionary function, see
[`SurplusReverserProtocol`](#surplusreverserprotocol) for more information
- `stderr: typing.TextIO = sys.stderr`
[TextIO-like object](https://docs.python.org/3/library/io.html#text-i-o)
@ -642,6 +787,87 @@ attributes
- `convert_to_type: ConversionResultTypeEnum = ConversionResultTypeEnum.SHAREABLE_TEXT`
what type to convert the query to
### `class SurplusDefaultGeocoding`
> [!IMPORTANT]
> this has replaced the now deprecated default geocoding functions, `default_geocoder()`
> and `default_reverser()`, in surplus 2.1.0 and later.
see [SurplusGeocoderProtocol](#surplusgeocoderprotocol) and
[SurplusReverserProtocol](#surplusreverserprotocol) for more information how to
implement a compliant custom geocoder functions.
[`dataclasses.dataclass`](https://docs.python.org/3/library/dataclasses.html) providing
the default geocoding functionality for surplus, via
[OpenStreetMap Nominatim](https://nominatim.openstreetmap.org/)
attributes
- `user_agent: str = default_fingerprint`
pass in a custom user agent here, else it will be the default
[fingerprinted user agent](#details-on-the-fingerprinted-user-agent)
example usage
```python
from surplus import surplus, Behaviour, SurplusDefaultGeocoding
geocoding = SurplusDefaultGeocoding("custom user agent")
geocoding.update_geocoding_functions() # not necessary but recommended
behaviour = Behaviour(
...,
geocoder=geocoding.geocoder,
reverser=geocoding.reverser
)
result = surplus("query", behaviour=behaviour)
...
```
methods
- [`def update_geocoding_functions(self) -> None: ...`](#surplusdefaultgeocodingupdate_geocoding_functions)
- [`def geocoder(self, place: str) -> Latlong: ...`](#surplusdefaultgeocodinggeocoder)
- [`def reverser(self, latlong: Latlong, level: int = 18) -> dict[str, Any]: ...`](#surplusdefaultgeocodingreverser)
#### `SurplusDefaultGeocoding.update_geocoding_functions()`
re-initialise the geocoding functions with the current user agent, also generate a new
user agent if not set properly
it is recommended to call this before using surplus as by default the geocoding functions
are uninitialised
- signature
```python
def update_geocoding_functions(self) -> None: ...
```
#### `SurplusDefaultGeocoding.geocoder()`
> [!WARNING]
> this function is primarily given to be passed into a [`Behaviour`](#class-behaviour)
> object, and is not meant to be called directly.
default geocoder for surplus
see [SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information on surplus
geocoder functions
#### `SurplusDefaultGeocoding.reverser()`
> [!WARNING]
> this function is primarily given to be passed into a [`Behaviour`](#class-behaviour)
> object, and is not meant to be called directly.
default reverser for surplus
see [SurplusReverserProtocol](#surplusreverserprotocol) for more information on surplus
reverser functions
### `class ConversionResultTypeEnum`
[enum.Enum](https://docs.python.org/3/library/enum.html)
@ -755,6 +981,11 @@ attributes
- `latitude: float`
- `longitude: float`
- `bounding_box: tuple[float, float, float, float] | None = None`
a four-tuple representing a bounding box, `(lat1, lat2, lon1, lon2)` or None.
the user does not need to enter this. the attribute is only used when shortening plus
codes, and would be supplied by the geocoding service during shortening.
methods
@ -762,7 +993,7 @@ methods
#### `Latlong.__str__()`
method that returns a comma-and-space-seperated string of `self.latitude` and
method that returns a comma-and-space-separated string of `self.latitude` and
`self.longitude`
- signature
@ -792,15 +1023,15 @@ methods
- signature
```python
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]:
def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
...
```
- arguments
- `geocoder: typing.Callable[[str], Latlong]`
name string to location function, must take in a string and return a
[`Latlong`](#class-latlong), exceptions are handled by the caller
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)[`[Latlong]`](#class-latlong)
@ -844,15 +1075,15 @@ exclusive method that returns a full-length Plus Code as a string
- signature
```python
def to_full_plus_code(self, geocoder: Callable[[str], Latlong]) -> Result[str]:
def to_full_plus_code(self, geocoder: SurplusGeocoderProtocol) -> Result[str]:
...
```
- arguments
- `geocoder: typing.Callable[[str], Latlong]`
name string to location function, must take in a string and return a
[`Latlong`](#class-latlong), exceptions are handled by the caller
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)`[str]`
@ -863,15 +1094,15 @@ method that returns a latitude-longitude coordinate pair
- signature
```python
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]:
def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
...
```
- arguments
- `geocoder: typing.Callable[[str], Latlong]`
name string to location function, must take in a string and return a
[`Latlong`](#class-latlong), exceptions are handled by the caller
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)[`[Latlong]`](#class-latlong)
@ -908,15 +1139,15 @@ method that returns a latitude-longitude coordinate pair
- signature
```python
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]:
def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
...
```
- arguments
- `geocoder: typing.Callable[[str], Latlong]`
name string to location function, must take in a string and return a
[`Latlong`](#class-latlong), exceptions are handled by the caller
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)[`[Latlong]`](#class-latlong)
@ -953,15 +1184,15 @@ method that returns a latitude-longitude coordinate pair
- signature
```python
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]:
def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
...
```
- arguments
- `geocoder: typing.Callable[[str], Latlong]`
name string to location function, must take in a string and return a
[`Latlong`](#class-latlong), exceptions are handled by the caller
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)[`[Latlong]`](#class-latlong)
@ -1014,35 +1245,90 @@ function that parses a query string into a query object
- returns [`Result`](#class-result)[`[Query]`](#query)
### `def default_geocoder()`
### `def generate_fingerprinted_user_agent()`
default geocoder for surplus, uses OpenStreetMap Nominatim
> [!NOTE]
> function is not used by surplus and not directly by the user, but is exposed for
> convenience being [Behaviour](#class-behaviour) objects.
> pass in a custom function to [Behaviour](#class-behaviour) to override the default reverser.
function that attempts to return a unique user agent string.
- signature
```python
def default_geocoder(place: str) -> Latlong:
```
```python
def generate_fingerprinted_user_agent() -> Result[str]:
```
### `def default_reverser()`
- returns [`Result[str]`](#class-result)
default reverser for surplus, uses OpenStreetMap Nominatim
this result will always have a valid value as erroneous results will have a
resulting value of `'surplus/<version> (generic-user)'`
> [!NOTE]
> function is not used by surplus and not directly by the user, but is exposed for
> convenience being [Behaviour](#class-behaviour) objects.
> pass in a custom function to [Behaviour](#class-behaviour) to override the default reverser.
valid results will have a value of `'surplus/<version> (<fingerprin hasht>)'`, where
the fingerprint hash is a 12 character hexadecimal string
- signature
#### details on the fingerprinted user agent
```python
def default_reverser(latlong: Latlong) -> dict[str, Any]:
```
**why do this in the first place?**
if too many people use surplus at the same time,
Nominatim will start to think it's just one person being greedy. so to prevent this,
surplus will try to generate a unique user agent string for each user through
fingerprinting.
at the time of writing, the pre-hashed fingerprint string is as follows:
```python
unique_info: str = f"{version}-{system_info}-{hostname}-{mac_address}"
```
it contains the following, in order, alongside an example:
1. `version` - the surplus version alongside a suffix, if any
```text
2.1.0-local
```
2. `system_info` - generic machine and operating system information
```text
Linux-6.5.0-locietta-WSL2-xanmod1-x86_64-with-glibc2.35
```
3. `hostname` - your computer's hostname
```text
mark
```
4. `mac_address` - your computer's mac address
```text
A9:36:3C:98:79:33
```
after hashing, this string becomes a 12 character hexadecimal string, as shown below:
```text
surplus/2.1.0-local (1fdbfa0b0cfb)
^^^^^^^^^^
this is the hashed result of unique_info
```
if at any time the retrieval of any of these four elements fail, surplus will just give
up and default to `'surplus/<version> (generic-user)'`.
if any of this seems weird to you, that's fine. pass in a custom user agent flag to
surplus with `-u` or `--user-agent` to override the default user agent, or override the
default user agent in your own code by passing in a custom user agent string to
[`Behaviour`](#class-behaviour).
```text
$ surplus --user_agent "a-shiny-custom-and-unique-user-agent" 77Q4+7X Austin, Texas, USA
...
```
```python
>>> from surplus import surplus, Behaviour
>>> surplus(..., Behaviour(user_agent="a-shiny-custom-and-unique-user-agent"))
...
```
## licence
@ -1053,21 +1339,15 @@ python module docstring.
however, direct dependencies of surplus are licensed under different, but still permissive
and open-source licences.
```text
geopy 2.4.0 Python Geocoding Toolbox
└── geographiclib >=1.52,<3
pluscodes 2022.1.3 Compute Plus Codes (Open Location Codes).
```
- [geopy](https://pypi.org/project/geopy/):
Python Geocoding Toolbox
MIT License
MIT Licence
- [geographiclib](https://pypi.org/project/geographiclib/):
The geodesic routines from GeographicLib
MIT License
MIT Licence
- [pluscodes](https://pypi.org/project/pluscodes/):
Compute Plus Codes (Open Location Codes)

451
playground.ipynb
View file

@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# surplus 2.0.0 playground notebook\n",
"# surplus 2.x.y playground notebook\n",
"\n",
"wrangling with environments for devbox users using codium/vs code:\n",
"\n",
@ -48,7 +48,9 @@
"source": [
"from surplus import PlusCodeQuery, LocalCodeQuery, LatlongQuery, StringQuery\n",
"from surplus import Latlong, Result\n",
"from surplus import default_geocoder, default_reverser"
"from surplus import SurplusDefaultGeocoding\n",
"\n",
"geocoding = SurplusDefaultGeocoding()"
]
},
{
@ -60,7 +62,7 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 3,
"metadata": {},
"outputs": [
{
@ -78,9 +80,9 @@
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/home/m/works/surplus/playground.ipynb Cell 5\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=7'>8</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\u001b[39mbool\u001b[39m(nom_result), \u001b[39mrepr\u001b[39m(nom_result\u001b[39m.\u001b[39merror), nom_result\u001b[39m.\u001b[39mget()))\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=8'>9</a>\u001b[0m \u001b[39mprint\u001b[39m(\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=9'>10</a>\u001b[0m \u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=10'>11</a>\u001b[0m \u001b[39mbool\u001b[39m(exc_result), \u001b[39mrepr\u001b[39m(exc_result\u001b[39m.\u001b[39merror), exc_result\u001b[39m.\u001b[39mcry(string\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m)\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=11'>12</a>\u001b[0m )\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=12'>13</a>\u001b[0m )\n\u001b[0;32m---> <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=13'>14</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\u001b[39mbool\u001b[39m(exc_result), \u001b[39mrepr\u001b[39m(exc_result\u001b[39m.\u001b[39merror), exc_result\u001b[39m.\u001b[39;49mget()))\n",
"File \u001b[0;32m~/works/surplus/surplus.py:247\u001b[0m, in \u001b[0;36mResult.get\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 245\u001b[0m \u001b[39m\"\"\"method that returns self.value if Result is non-erroneous else raises error\"\"\"\u001b[39;00m\n\u001b[1;32m 246\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39merror, \u001b[39mBaseException\u001b[39;00m):\n\u001b[0;32m--> 247\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39merror\n\u001b[1;32m 248\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mvalue\n",
"\u001b[1;32m/home/m/works/surplus/playground.ipynb Cell 5\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0'>1</a>\u001b[0m nom_result \u001b[39m=\u001b[39m Result[\u001b[39mint\u001b[39m](\u001b[39m3\u001b[39m)\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m----> <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3'>4</a>\u001b[0m \u001b[39m1\u001b[39;49m \u001b[39m/\u001b[39;49m \u001b[39m0\u001b[39;49m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4'>5</a>\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mException\u001b[39;00m \u001b[39mas\u001b[39;00m exc:\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5'>6</a>\u001b[0m exc_result \u001b[39m=\u001b[39m Result[\u001b[39mint\u001b[39m](\u001b[39m-\u001b[39m\u001b[39m1\u001b[39m, error\u001b[39m=\u001b[39mexc)\n",
"\u001b[1;32m/home/m/works/surplus/playground.ipynb Cell 5\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=7'>8</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\u001b[39mbool\u001b[39m(nom_result), \u001b[39mrepr\u001b[39m(nom_result\u001b[39m.\u001b[39merror), nom_result\u001b[39m.\u001b[39mget()))\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=8'>9</a>\u001b[0m \u001b[39mprint\u001b[39m(\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=9'>10</a>\u001b[0m \u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=10'>11</a>\u001b[0m \u001b[39mbool\u001b[39m(exc_result), \u001b[39mrepr\u001b[39m(exc_result\u001b[39m.\u001b[39merror), exc_result\u001b[39m.\u001b[39mcry(string\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m)\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=11'>12</a>\u001b[0m )\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=12'>13</a>\u001b[0m )\n\u001b[0;32m---> <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=13'>14</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\u001b[39mbool\u001b[39m(exc_result), \u001b[39mrepr\u001b[39m(exc_result\u001b[39m.\u001b[39merror), exc_result\u001b[39m.\u001b[39;49mget()))\n",
"File \u001b[0;32m~/works/surplus/surplus/surplus.py:270\u001b[0m, in \u001b[0;36mResult.get\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 268\u001b[0m \u001b[39m\"\"\"method that returns self.value if Result is non-erroneous else raises error\"\"\"\u001b[39;00m\n\u001b[1;32m 269\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39merror, \u001b[39mBaseException\u001b[39;00m):\n\u001b[0;32m--> 270\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39merror\n\u001b[1;32m 271\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mvalue\n",
"\u001b[1;32m/home/m/works/surplus/playground.ipynb Cell 5\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0'>1</a>\u001b[0m nom_result \u001b[39m=\u001b[39m Result[\u001b[39mint\u001b[39m](\u001b[39m3\u001b[39m)\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m----> <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3'>4</a>\u001b[0m \u001b[39m1\u001b[39;49m \u001b[39m/\u001b[39;49m \u001b[39m0\u001b[39;49m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4'>5</a>\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mException\u001b[39;00m \u001b[39mas\u001b[39;00m exc:\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5'>6</a>\u001b[0m exc_result \u001b[39m=\u001b[39m Result[\u001b[39mint\u001b[39m](\u001b[39m-\u001b[39m\u001b[39m1\u001b[39m, error\u001b[39m=\u001b[39mexc)\n",
"\u001b[0;31mZeroDivisionError\u001b[0m: division by zero"
]
}
@ -111,27 +113,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Result(value=Latlong(latitude=1.3336875, longitude=103.7746875), error=None)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"PlusCodeQuery(code=\"6PH58QMF+FV\").to_lat_long_coord(geocoder=default_geocoder)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {},
"outputs": [
{
@ -146,16 +128,12 @@
}
],
"source": [
"plus_code = LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_full_plus_code(\n",
" geocoder=default_geocoder\n",
")\n",
"\n",
"PlusCodeQuery(code=plus_code.get()).to_lat_long_coord(geocoder=default_geocoder)"
"PlusCodeQuery(code=\"6PH58QMF+FV\").to_lat_long_coord(geocoder=geocoding.geocoder)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {},
"outputs": [
{
@ -170,20 +148,22 @@
}
],
"source": [
"LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_lat_long_coord(\n",
" geocoder=default_geocoder\n",
")"
"plus_code = LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_full_plus_code(\n",
" geocoder=geocoding.geocoder\n",
")\n",
"\n",
"PlusCodeQuery(code=plus_code.get()).to_lat_long_coord(geocoder=geocoding.geocoder)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Result(value=Latlong(latitude=1.33318835, longitude=103.77461234638255), error=None)"
"Result(value=Latlong(latitude=1.3336875, longitude=103.7746875), error=None)"
]
},
"execution_count": 6,
@ -192,14 +172,14 @@
}
],
"source": [
"LatlongQuery(\n",
" latlong=Latlong(latitude=1.33318835, longitude=103.77461234638255)\n",
").to_lat_long_coord(geocoder=default_geocoder)"
"LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_lat_long_coord(\n",
" geocoder=geocoding.geocoder\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"metadata": {},
"outputs": [
{
@ -214,7 +194,29 @@
}
],
"source": [
"StringQuery(query=\"Ngee Ann Polytechnic\").to_lat_long_coord(geocoder=default_geocoder)"
"LatlongQuery(\n",
" latlong=Latlong(latitude=1.33318835, longitude=103.77461234638255)\n",
").to_lat_long_coord(geocoder=geocoding.geocoder)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Result(value=Latlong(latitude=1.33318835, longitude=103.77461234638255), error=None)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"StringQuery(query=\"Ngee Ann Polytechnic\").to_lat_long_coord(geocoder=geocoding.geocoder)"
]
},
{
@ -282,7 +284,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"metadata": {},
"outputs": [
{
@ -298,6 +300,7 @@
" 'house_number': '535',\n",
" 'latitude': 1.33318835,\n",
" 'longitude': 103.77461234638255,\n",
" 'neighbourhood': 'Ewart Park',\n",
" 'postcode': '599489',\n",
" 'raw': {'address': {'ISO3166-2-lvl6': 'SG-03',\n",
" 'amenity': 'Ngee Ann Polytechnic',\n",
@ -306,6 +309,7 @@
" 'country_code': 'sg',\n",
" 'county': 'Northwest',\n",
" 'house_number': '535',\n",
" 'neighbourhood': 'Ewart Park',\n",
" 'postcode': '599489',\n",
" 'road': 'Clementi Road',\n",
" 'suburb': 'Bukit Timah'},\n",
@ -315,8 +319,9 @@
" '103.7701481',\n",
" '103.7783945'],\n",
" 'class': 'amenity',\n",
" 'display_name': 'Ngee Ann Polytechnic, 535, Clementi Road, Bukit '\n",
" 'Timah, Singapore, Northwest, 599489, Singapore',\n",
" 'display_name': 'Ngee Ann Polytechnic, 535, Clementi Road, Ewart '\n",
" 'Park, Bukit Timah, Singapore, Northwest, 599489, '\n",
" 'Singapore',\n",
" 'importance': 0.34662169301918117,\n",
" 'lat': '1.33318835',\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. '\n",
@ -325,7 +330,7 @@
" 'name': 'Ngee Ann Polytechnic',\n",
" 'osm_id': 2535118,\n",
" 'osm_type': 'relation',\n",
" 'place_id': 297946059,\n",
" 'place_id': 250910125,\n",
" 'place_rank': 30,\n",
" 'type': 'university'},\n",
" 'road': 'Clementi Road',\n",
@ -337,15 +342,363 @@
"import pprint\n",
"\n",
"latlong = LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_lat_long_coord(\n",
" default_geocoder\n",
" geocoder=geocoding.geocoder\n",
")\n",
"if not latlong:\n",
" latlong.cry()\n",
"\n",
"else:\n",
" location = default_reverser(latlong.get())\n",
" location = geocoding.reverser(latlong.get())\n",
" pprint.pprint(location)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.1.0: adventures in of shortening global/full Plus Codes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### testing rate-limited and cached default geocoding functions"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"test_geocoding = SurplusDefaultGeocoding(user_agent=\"surplus/playground\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1\n",
"2\n",
"3\n",
"4\n",
"5\n",
"\n",
"1\n",
"2\n",
"3\n",
"4\n",
"5\n",
"\n",
"3.1107698050s\t->\t0.0000886890s\t\t(-3.1106811160002508s)\n"
]
}
],
"source": [
"from timeit import timeit\n",
"\n",
"\n",
"test_stmt = \"\"\"\\\n",
"print(1)\n",
"test_geocoding.geocoder(\"Wisma Atria\") # instant\n",
"print(2)\n",
"test_geocoding.geocoder(\"Temasek Polytechnic\") # after 1 second\n",
"print(3)\n",
"location = test_geocoding.geocoder(\"Ngee Ann Polytechnic\") # after 1 second\n",
"print(4)\n",
"test_geocoding.reverser(f\"{location.latitude}, {location.longitude}\") # instant\n",
"print(5)\n",
"test_geocoding.reverser(f\"{location.latitude}, {location.longitude}\") # instant (cached)\n",
"print()\n",
"\"\"\"\n",
"\n",
"time_cold_call = timeit(test_stmt, globals=globals(), number=1) # expecting 3-4 seconds\n",
"time_2nd_call = timeit(test_stmt, globals=globals(), number=1) # should be instant\n",
"\n",
"print(\n",
" f\"{time_cold_call:.10f}s\\t->\\t{time_2nd_call:.10f}s\\t\\t({time_2nd_call - time_cold_call}s)\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### reversing the query latlong and using the address information to form a locality"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"level = 13"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"St Lucia, St Lucia, Queensland, Australia\n",
"Austin, Travis County, Texas, United States\n"
]
}
],
"source": [
"(\n",
" au_response := geocoding.reverser(\n",
" (\n",
" au_target := (\n",
" LocalCodeQuery(\n",
" \"G227+XF\", \"St Lucia, Queensland, Australia\"\n",
" ).to_lat_long_coord(geocoding.geocoder)\n",
" )\n",
" ).get(),\n",
" level=level,\n",
" )\n",
")\n",
"\n",
"au_locality = f\"{au_response['suburb']}, {au_response['city_district']}, {au_response['state']}, {au_response['country']}\"\n",
"print(au_locality)\n",
"\n",
"(\n",
" us_response := geocoding.reverser(\n",
" (\n",
" us_target := (\n",
" LocalCodeQuery(\"77Q4+7X\", \"Austin, Texas, USA\").to_lat_long_coord(\n",
" geocoding.geocoder\n",
" )\n",
" )\n",
" ).get(),\n",
" level=level,\n",
" )\n",
")\n",
"\n",
"us_locality = f\"{us_response['city']}, {us_response['county']}, {us_response['state']}, {us_response['country']}\"\n",
"print(us_locality)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### getting boundary boxes"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'addresstype': 'suburb',\n",
" 'boundingbox': ['-27.5187362', '-27.4787362', '152.9881642', '153.0281642'],\n",
" 'class': 'place',\n",
" 'display_name': 'St Lucia, Brisbane City, Queensland, 4072, Australia',\n",
" 'importance': 0.27501,\n",
" 'lat': '-27.4987362',\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. '\n",
" 'http://osm.org/copyright',\n",
" 'lon': '153.0081642',\n",
" 'name': 'St Lucia',\n",
" 'osm_id': 88800268,\n",
" 'osm_type': 'node',\n",
" 'place_id': 54477898,\n",
" 'place_rank': 19,\n",
" 'type': 'suburb'}\n",
"\n",
"Latlong(latitude=-27.4987362, longitude=153.0081642, bounding_box=[-27.5187362, -27.4787362, 152.9881642, 153.0281642])\n"
]
}
],
"source": [
"from geopy.geocoders import Nominatim\n",
"from pprint import pprint\n",
"\n",
"target_query: Result[Latlong] = au_target\n",
"target_locality: str = au_locality\n",
"\n",
"raw_geocoding = Nominatim(user_agent=\"surplus/playground\")\n",
"latlong = raw_geocoding.geocode(target_locality)\n",
"pprint(latlong.raw)\n",
"print()\n",
"\n",
"# done: now implmented in surplus as surplus.Latlong.bounding_box\n",
"locality_latlong = geocoding.geocoder(target_locality)\n",
"pprint(locality_latlong)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[-27.5187362, -27.4787362, 152.9881642, 153.0281642]\n",
"(True, True, True, True)\n",
"(True, True, True, True)\n"
]
}
],
"source": [
"# based on <https://github.com/google/open-location-code/wiki/Guidance-for-shortening-codes>\n",
"\n",
"target_latlong = target_query.get()\n",
"if locality_latlong.bounding_box is None:\n",
" ... # raise some error\n",
"\n",
"print(locality_latlong.bounding_box)\n",
"check1 = (\n",
" # The center point of the feature is within 0.4 degrees latitude and 0.4 degrees longitude\n",
" (\n",
" (target_latlong.latitude - 0.4)\n",
" <= locality_latlong.latitude\n",
" <= (target_latlong.latitude + 0.4)\n",
" ),\n",
" (\n",
" (target_latlong.longitude - 0.4)\n",
" <= locality_latlong.longitude\n",
" <= (target_latlong.longitude + 0.4)\n",
" ),\n",
" # The bounding box of the feature is less than 0.8 degrees high and wide.\n",
" abs(locality_latlong.bounding_box[0] - locality_latlong.bounding_box[1]) < 0.8,\n",
" abs(locality_latlong.bounding_box[2] - locality_latlong.bounding_box[3]) < 0.8,\n",
")\n",
"\n",
"\n",
"check2 = (\n",
" # The center point of the feature is within 0.4 degrees latitude and 0.4 degrees longitude\n",
" (\n",
" (target_latlong.latitude - 8)\n",
" <= locality_latlong.latitude\n",
" <= (target_latlong.latitude + 8)\n",
" ),\n",
" (\n",
" (target_latlong.longitude - 8)\n",
" <= locality_latlong.longitude\n",
" <= (target_latlong.longitude + 8)\n",
" ),\n",
" # The bounding box of the feature is less than 0.8 degrees high and wide.\n",
" abs(locality_latlong.bounding_box[0] - locality_latlong.bounding_box[1]) < 16,\n",
" abs(locality_latlong.bounding_box[2] - locality_latlong.bounding_box[3]) < 16,\n",
")\n",
"\n",
"print(check1)\n",
"print(check2)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"G227+XF St Lucia, St Lucia, Queensland, Australia\n"
]
}
],
"source": [
"from pluscodes import encode\n",
"\n",
"target_plus_code = encode(\n",
" lat=target_latlong.latitude, lon=target_latlong.longitude, code_length=10\n",
")\n",
"portion_plus_code = \"\"\n",
"\n",
"if check1:\n",
" portion_plus_code = target_plus_code[4:]\n",
" print(portion_plus_code, target_locality)\n",
"\n",
"elif check2:\n",
" portion_plus_code = target_plus_code[2:]\n",
" print(portion_plus_code, target_locality)\n",
"\n",
"else:\n",
" print(\n",
" \"info: could not determine a suitable geographical feature to use as locality for shortening.\"\n",
" )\n",
" print(plus_code)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## machine fingerprinting attempt\n",
"\n",
"because of nominatim's acceptable usage policy \n",
"<https://operations.osmfoundation.org/policies/nominatim/>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from hashlib import shake_256 as _hashlib_shake_256\n",
"from platform import platform as _platform_platform\n",
"from socket import gethostname as _socket_gethostname\n",
"from uuid import getnode as _uuid_getnode\n",
"from surplus import VERSION, VERSION_SUFFIX\n",
"\n",
"\n",
"def generate_fingerprinted_user_agent() -> Result[str]:\n",
" \"\"\"\n",
" function that attempts to return a unique user agent string.\n",
"\n",
" returns Result[str]\n",
" this result will always have a valid value as erroneous results will have a\n",
" resulting value of 'surplus/<version>/generic-user'\n",
" valid results will have a value of 'surplus/<version>/<fingerprint>', where\n",
" fingerprint is a 12 character hexadecimal string\n",
" \"\"\"\n",
" version: str = \".\".join([str(v) for v in VERSION]) + VERSION_SUFFIX\n",
"\n",
" try:\n",
" system_info: str = _platform_platform()\n",
" hostname: str = _socket_gethostname()\n",
" mac_address: str = \":\".join(\n",
" [\n",
" \"{:02x}\".format((_uuid_getnode() >> elements) & 0xFF)\n",
" for elements in range(0, 2 * 6, 2)\n",
" ][::-1]\n",
" )\n",
" unique_info: str = f\"{version}-{system_info}-{hostname}-{mac_address}\"\n",
"\n",
" print(f\"{version=}\")\n",
" print(f\"{system_info=}\")\n",
" print(f\"{hostname=}\")\n",
" print(f\"{mac_address=}\")\n",
"\n",
" except Exception as exc:\n",
" return Result[str](f\"surplus/{version} (generic-user)\", error=exc)\n",
"\n",
" fingerprint: str = _hashlib_shake_256(unique_info.encode()).hexdigest(5)\n",
"\n",
" return Result[str](f\"surplus/{version} ({fingerprint})\")"
]
}
],
"metadata": {

6
poetry.lock generated
View file

@ -14,14 +14,14 @@ files = [
[[package]]
name = "asttokens"
version = "2.3.0"
version = "2.4.0"
description = "Annotate AST trees with source code positions"
category = "dev"
optional = false
python-versions = "*"
files = [
{file = "asttokens-2.3.0-py2.py3-none-any.whl", hash = "sha256:bef1a51bc256d349e9f94e7e40e44b705ed1162f55294220dd561d24583d9877"},
{file = "asttokens-2.3.0.tar.gz", hash = "sha256:2552a88626aaa7f0f299f871479fc755bd4e7c11e89078965e928fb7bb9a6afe"},
{file = "asttokens-2.4.0-py2.py3-none-any.whl", hash = "sha256:cf8fc9e61a86461aa9fb161a14a0841a03c405fa829ac6b202670b3495d2ce69"},
{file = "asttokens-2.4.0.tar.gz", hash = "sha256:2e0171b991b2c959acc6c49318049236844a5da1d65ba2672c4880c1c894834e"},
]
[package.dependencies]

2
pyproject.toml
View file

@ -1,6 +1,6 @@
[tool.poetry]
name = "surplus"
version = "2.0.1"
version = "2.1.0"
description = "Python script to convert Google Maps Plus Codes to iOS Shortcuts-like shareable text."
authors = ["Mark Joshwel <mark@joshwel.co>"]
license = "Unlicense"

90
releaser.py Normal file
View file

@ -0,0 +1,90 @@
"""
surplus: Google Maps Plus Code to iOS Shortcuts-like shareable text
-------------------------------------------------------------------
by mark <mark@joshwel.co>
This is free and unencumbered software released into the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.
In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
For more information, please refer to <http://unlicense.org/>
"""
from datetime import datetime, timedelta, timezone
from pathlib import Path
from subprocess import run
# NOTE: change this if surplus has moved
path_surplus = Path(__file__).parent.joinpath("./surplus/surplus.py")
build_time = datetime.now(timezone(timedelta(hours=8))) # using SGT
insert_build_branch: str = run(
"git rev-parse --abbrev-ref HEAD",
capture_output=True,
text=True,
shell=True,
).stdout.strip("\n")
insert_build_commit: str = run(
"git rev-parse HEAD",
capture_output=True,
text=True,
shell=True,
).stdout.strip("\n")
insert_build_datetime: str = repr(build_time).replace("datetime.", "")
# NOTE: change this if the respective lines in surplus.py have changed
targets: list[tuple[str, str]] = [
(
'VERSION_SUFFIX: Final[str] = "-local"',
'VERSION_SUFFIX: Final[str] = ""',
),
(
'BUILD_BRANCH: Final[str] = "future"',
f'BUILD_BRANCH: Final[str] = "{insert_build_branch}"',
),
(
'BUILD_COMMIT: Final[str] = "latest"',
f'BUILD_COMMIT: Final[str] = "{insert_build_commit}"',
),
(
"BUILD_DATETIME: Final[datetime] = datetime.now(timezone(timedelta(hours=8))) # using SGT",
f"BUILD_DATETIME: Final[datetime] = {insert_build_datetime}",
),
]
def main() -> int:
assert path_surplus.is_file() and path_surplus.exists(), f"{path_surplus} not found"
source_surplus: str = path_surplus.read_text(encoding="utf-8")
for old, new in targets:
print(f"new: {new}\nold: {old}\n")
source_surplus = source_surplus.replace(old, new)
path_surplus.write_text(source_surplus, encoding="utf-8")
return 0
if __name__ == "__main__":
exit(main())

17
surplus/__init__.py
View file

@ -32,7 +32,14 @@ For more information, please refer to <http://unlicense.org/>
# surplus was and would've been a single-file module, but typing is in the way :(
# https://github.com/python/typing/issues/1333
from .surplus import default_geocoder # deprecated, emulation function
from .surplus import default_reverser # deprecated, emulation function
from .surplus import (
BUILD_BRANCH,
BUILD_COMMIT,
BUILD_DATETIME,
CONNECTION_MAX_RETRIES,
CONNECTION_WAIT_SECONDS,
EMPTY_LATLONG,
SHAREABLE_TEXT_LINE_0_KEYS,
SHAREABLE_TEXT_LINE_1_KEYS,
@ -41,9 +48,10 @@ from .surplus import (
SHAREABLE_TEXT_LINE_4_KEYS,
SHAREABLE_TEXT_LINE_5_KEYS,
SHAREABLE_TEXT_LINE_6_KEYS,
SHAREABLE_TEXT_LOCALITY,
SHAREABLE_TEXT_NAMES,
USER_AGENT,
VERSION,
VERSION_SUFFIX,
Behaviour,
ConversionResultTypeEnum,
EmptyQueryError,
@ -59,11 +67,12 @@ from .surplus import (
Result,
ResultType,
StringQuery,
SurplusDefaultGeocoding,
SurplusException,
UnavailableFeatureError,
SurplusGeocoderProtocol,
SurplusReverserProtocol,
cli,
default_geocoder,
default_reverser,
generate_fingerprinted_user_agent,
handle_args,
parse_query,
surplus,

880
surplus/surplus.py
View file

File diff suppressed because it is too large Load diff

44
test.py
View file

@ -3,7 +3,7 @@
"""
surplus test runner
-------------------
by mark <mark@joshwel.co> and contributors
by mark <mark@joshwel.co>
This is free and unencumbered software released into the public domain.
@ -100,22 +100,38 @@ tests: list[ContinuityTest] = [
),
ContinuityTest(
query="Ngee Ann Polytechnic, Singapore",
expected=(
"Ngee Ann Polytechnic\n"
"535 Clementi Road\n"
"Bukit Timah\n"
"599489\n"
"Northwest, Singapore"
),
expected=[
(
"Ngee Ann Polytechnic\n"
"535 Clementi Road\n"
"Bukit Timah\n"
"599489\n"
"Northwest, Singapore"
)
],
),
ContinuityTest(
query="1.3521, 103.8198",
expected=(
"MacRitchie Nature Trail\n"
"Central Water Catchment\n"
"574325\n"
"Central, Singapore"
),
expected=[
(
"MacRitchie Nature Trail\n"
"Central Water Catchment\n"
"574325\n"
"Central, Singapore"
)
],
),
ContinuityTest(
query="8WWJ+4P, Singapore", # a comma!
expected=[
(
"Temasek Polytechnic\n"
"21 Tampines Avenue 1\n"
"Tampines West\n"
"529757\n"
"Northeast, Singapore"
)
],
),
]