mark/surplus
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

s+,docs,tests: many

Browse source 
- s+: local code conversion
- s+: fingerprinted user agents + override arg
- s+: rate limited default_ geocoding functions
- s+: gecoding function protocols
- docs: update api
- tests: fix np and macritchie expecteds
This commit is contained in:
Mark Joshwel 2023-09-05 17:34:28 +00:00
parent f6de01a029
commit fda37f413f
6 changed files with 969 additions and 243 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

410
README.md
View file

@ -22,6 +22,7 @@ to iOS Shortcuts-like shareable text.
- [what counts as "incorrect"](#what-counts-as-incorrect) - [what counts as "incorrect"](#what-counts-as-incorrect)
- [output technical details](#the-technical-details-of-surpluss-output) - [output technical details](#the-technical-details-of-surpluss-output)
- [api reference](#api-reference) - [api reference](#api-reference)
- [details on the fingerprinted user agent](#details-on-the-fingerprinted-user-agent)
- [licence](#licence) - [licence](#licence)
```text ```text
@ -62,27 +63,29 @@ see [licence](#licence) for licensing information.
### command-line usage ### command-line usage
```text ```text
usage: surplus [-h] [-d] [-v] [-c {pluscode,localcode,latlong,string}] usage: surplus [-h] [-d] [-v] [-c {pluscode,localcode,latlong,sharetext}]
[-u USER_AGENT]
[query ...] [query ...]
Google Maps Plus Code to iOS Shortcuts-like shareable text Google Maps Plus Code to iOS Shortcuts-like shareable text
positional arguments: positional arguments:
query full-length Plus Code (6PH58QMF+FX), query full-length Plus Code (6PH58QMF+FX), shortened
shortened Plus Code/'local code' (8QMF+FX Singapore), Plus Code/'local code' (8QMF+FX Singapore),
latlong (1.3336875, 103.7749375), latlong (1.3336875, 103.7749375), string query
string query (e.g., 'Wisma Atria'), (e.g., 'Wisma Atria'), or '-' to read from stdin
or '-' to read from stdin
options: options:
-h, --help show this help message and exit -h, --help show this help message and exit
-d, --debug prints lat, long and reverser response dict to -d, --debug prints lat, long and reverser response dict to
stderr stderr
-v, --version prints version information to stderr and exits -v, --version prints version information to stderr and exits
-c {pluscode,localcode,latlong,sharetext}, -c {pluscode,localcode,latlong,sharetext}, --convert-to {pluscode,localcode,latlong,sharetext}
--convert-to {pluscode,localcode,latlong,sharetext}
converts query a specific output type, defaults converts query a specific output type, defaults
to 'sharetext' to 'sharetext'
-u USER_AGENT, --user-agent USER_AGENT
user agent string to use for geocoding service,
defaults to fingerprinted user agent string
``` ```
### example api usage ### example api usage
@ -189,7 +192,7 @@ and do the following:
function, which by default is OpenStreetMap Nominatim. function, which by default is OpenStreetMap Nominatim.
(_don't know what the above means? then you are using the default reverser._) (_don't know what the above means? then you are using the default reverser._)
also look at the [what counts as "incorrect"](#what-counts-as-incorrect) section also look at the ['what counts as "incorrect"'](#what-counts-as-incorrect) section
before moving on. before moving on.
2. include the erroneous query. 2. include the erroneous query.
@ -263,12 +266,13 @@ of incorrect outputs.
```text ```text
$ s+ --debug 8QJF+RP Singapore $ s+ --debug 8QJF+RP Singapore
surplus version 2.1.0, debug mode surplus version 2.1.0, debug mode (latest@future, Tue 05 Sep 2023 23:38:59 +0800)
debug: parse_query: behaviour.query=['8QJF+RP', 'Singapore'] debug: parse_query: behaviour.query=['8QJF+RP', 'Singapore']
debug: _match_plus_code: portion_plus_code='8QJF+RP', portion_locality='Singapore' debug: _match_plus_code: portion_plus_code='8QJF+RP', portion_locality='Singapore'
debug: cli: query=Result(value=LocalCodeQuery(code='8QJF+RP', locality='Singapore'), error=None) debug: cli: query=Result(value=LocalCodeQuery(code='8QJF+RP', locality='Singapore'), error=None)
debug: cli: latlong.get()=Latlong(latitude=1.3320625, longitude=103.7743125) debug: cli: latlong_result.get()=Latlong(latitude=1.3320625, longitude=103.7743125)
debug: cli: location={'amenity': 'Ngee Ann Polytechnic', 'house_number': '535', 'road': 'Clementi Road', 'suburb': 'Bukit Timah', 'city': 'Singapore', 'county': 'Northwest', 'ISO3166-2-lvl6': 'SG-03', 'postcode': '599489', 'country': 'Singapore', 'country_code': 'sg', 'raw': "{...}", 'latitude': '1.33318835', 'longitude': '103.77461234638255'} debug: cli: behaviour.user_agent='surplus/2.1.0-local (1fdbfa0b0cfb)'
debug: cli: location={...}
debug: _generate_text: seen_names=['Ngee Ann Polytechnic', 'Clementi Road'] debug: _generate_text: seen_names=['Ngee Ann Polytechnic', 'Clementi Road']
debug: _generate_text_line: [True] -> True -------- 'Ngee Ann Polytechnic' debug: _generate_text_line: [True] -> True -------- 'Ngee Ann Polytechnic'
debug: _generate_text_line: [True] -> True -------- '535' debug: _generate_text_line: [True] -> True -------- '535'
@ -329,7 +333,7 @@ variables
this variable is displayed to show what query type [`parse_query()`](#def-parse_query) has this variable is displayed to show what query type [`parse_query()`](#def-parse_query) has
recognised, and if there were any errors during query parsing recognised, and if there were any errors during query parsing
- **expression `latlong.get()=`** - **expression `latlong_result.get()=`**
(_only shown if the query is a plus code_) (_only shown if the query is a plus code_)
@ -340,12 +344,12 @@ variables
the response dictionary from the reverser function passed to the response dictionary from the reverser function passed to
[`surplus()`](#def-surplus) [`surplus()`](#def-surplus)
for more information on the reverser function, see [`Behaviour`](#class-behaviour) and for more information on the reverser function, see
[`default_reverser`](#def-default_reverser) [`SurplusReverserProtocol`](#surplusreverserprotocol)
- **variable `seen_names`** - **variable `seen_names`**
a list of unique important names found in certain nominatim keys used in final output a list of unique important names found in certain Nominatim keys used in final output
lines 0-3 lines 0-3
- **`_generate_text_line` seen name checks** - **`_generate_text_line` seen name checks**
@ -524,7 +528,13 @@ line breakdown of shareable text output, accompanied by their Nominatim keys:
- [types](#types) - [types](#types)
- [`Query`](#query) - [`Query`](#query)
- [`ResultType`](#resulttype) - [`ResultType`](#resulttype)
- [`SurplusGeocoderProtocol`](#surplusgeocoderprotocol)
- [`SurplusReverserProtocol`](#surplusreverserprotocol)
- [`class Behaviour`](#class-behaviour) - [`class Behaviour`](#class-behaviour)
- [`class SurplusDefaultGeocoding`](#class-surplusdefaultgeocoding)
- [`SurplusDefaultGeocoding.update_geocoding_functions()`](#surplusdefaultgeocodingupdate_geocoding_functions)
- [`SurplusDefaultGeocoding.geocoder()`](#surplusdefaultgeocodinggeocoder)
- [`SurplusDefaultGeocoding.reverser()`](#surplusdefaultgeocodingreverser)
- [`class ConversionResultTypeEnum`](#class-conversionresulttypeenum) - [`class ConversionResultTypeEnum`](#class-conversionresulttypeenum)
- [`class Result`](#class-result) - [`class Result`](#class-result)
- [`Result.__bool__()`](#result__bool__) - [`Result.__bool__()`](#result__bool__)
@ -547,8 +557,8 @@ line breakdown of shareable text output, accompanied by their Nominatim keys:
- [`StringQuery.__str__()`](#stringquery__str__) - [`StringQuery.__str__()`](#stringquery__str__)
- [`def surplus()`](#def-surplus) - [`def surplus()`](#def-surplus)
- [`def parse_query()`](#def-parse_query) - [`def parse_query()`](#def-parse_query)
- [`def default_geocoder()`](#def-default_geocoder) - [`def generate_fingerprinted_user_agent`](#def-generate_fingerprinted_user_agent)
- [`def default_reverser()`](#def-default_reverser) - [details on the fingerprinted user agent](#details-on-the-fingerprinted-user-agent)
### constants ### constants
@ -557,30 +567,40 @@ line breakdown of shareable text output, accompanied by their Nominatim keys:
a tuple of integers representing the version of surplus, in the format a tuple of integers representing the version of surplus, in the format
`[major, minor, patch]` `[major, minor, patch]`
- `VERSION_SUFFIX: Final[str]` - `VERSION_SUFFIX: typing.Final[str]`
`BUILD_BRANCH: Final[str]` `BUILD_BRANCH: typing.Final[str]`
`BUILD_COMMIT: Final[str]` `BUILD_COMMIT: typing.Final[str]`
`BUILD_DATETIME: Final[datetime]` `BUILD_DATETIME: typing.Final[datetime]`
string and a [datetime.datetime](https://docs.python.org/3/library/datetime.html) object string and a [datetime.datetime](https://docs.python.org/3/library/datetime.html) object
containing version and build information, set by [releaser.py](releaser.py) containing version and build information, set by [releaser.py](releaser.py)
- `SHAREABLE_TEXT_LINE_0_KEYS: tuple[str, ...]` - `CONNECTION_MAX_RETRIES: int = 9`
`SHAREABLE_TEXT_LINE_1_KEYS: tuple[str, ...]` `CONNECTION_WAIT_SECONDS: int = 10`
`SHAREABLE_TEXT_LINE_2_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_3_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_4_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_5_KEYS: tuple[str, ...]`
`SHAREABLE_TEXT_LINE_6_KEYS: tuple[str, ...]`
a tuple of strings containing nominatim keys used in shareable text line 0-6 defines if and how many times to retry a connection, alongside how many seconds to wait
in between tries, for Nominatim
- `SHAREABLE_TEXT_NAMES: tuple[str, ...]` > [!NOTE]
> this constant only affects the default surplus Nominatim geocoding functions. custom
> functions do not read from this, unless deliberately programmed to do so
a tuple of strings containing nominatim keys used in shareable text line 0-2 and - `SHAREABLE_TEXT_LINE_0_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_1_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_2_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_3_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_4_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_5_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_6_KEYS: typing.Final[tuple[str, ...]]`
a tuple of strings containing Nominatim keys used in shareable text line 0-6
- `SHAREABLE_TEXT_NAMES: typing.Final[tuple[str, ...]]`
a tuple of strings containing Nominatim keys used in shareable text line 0-2 and
special keys in line 3 special keys in line 3
- `EMPTY_LATLONG: Latlong` - `EMPTY_LATLONG: typing.Final[Latlong]`
a constant for an empty latlong coordinate, with latitude and longitude set to 0.0 a constant for an empty latlong coordinate, with latitude and longitude set to 0.0
### exception classes ### exception classes
@ -618,6 +638,88 @@ ResultType = TypeVar("ResultType")
[generic type](https://docs.python.org/3/library/typing.html#generics) used by [generic type](https://docs.python.org/3/library/typing.html#generics) used by
[`Result`](#class-result) [`Result`](#class-result)
#### `SurplusGeocoderProtocol`
[typing_extensions.Protocol](https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols)
class for documentation and static type checking of surplus geocoder functions
- **signature and conforming function signature**
```python
class SurplusGeocoderProtocol(Protocol):
def __call__(self, place: str) -> Latlong:
...
```
functions that conform to this protocol should have the following signature:
```python
def example(place: str) -> Latlong: ...
```
- **information on conforming functions**
function takes in a location name as a string, and returns a [Latlong](#class-latlong)
function can and should be be
[`functools.lru_cache()`-wrapped](https://docs.python.org/3/library/functools.html#functools.lru_cache)
if the geocoding service asks for caching
exceptions are handled by the caller
#### `SurplusReverserProtocol`
[typing_extensions.Protocol](https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols)
class for documentation and static type checking of surplus reverser functions
- **signature and conforming function signature**
```python
class SurplusReverserProtocol(Protocol):
def __call__(self, latlong: Latlong, level: int = 18) -> dict[str, Any]:
...
```
functions that conform to this protocol should have the following signature:
```python
def example(self, latlong: Latlong, level: int = 18) -> dict[str, Any]: ...
```
- **information on conforming functions**
function takes in a [Latlong](#class-latlong) object and return a dictionary with [`SHAREABLE_TEXT_LINE_*_KEYS`](#constants) keys at the dictionaries' top-level.
keys are used to access address information.
function should also take in an int representing the level of detail for the returned
address, 0-18 (country-level to building), inclusive. should default to 18.
keys for latitude, longitude and an iso3166-2 (or closest equivalent) should also be
included at the dictionaries top level as the keys `latitude`, `longitude` and
`ISO3166-2` (non-case sensitive, or at least something starting with `ISO3166`)
respectively.
```python
{
'ISO3166-2-lvl6': 'SG-03',
'amenity': 'Ngee Ann Polytechnic',
...
'country': 'Singapore',
'latitude': 1.33318835,
'longitude': 103.77461234638255,
'postcode': '599489',
'raw': {...},
}
```
function can and should be
[`functools.lru_cache()`-wrapped](https://docs.python.org/3/library/functools.html#functools.lru_cache)
if the geocoding service asks for caching
see the [playground notebook](/playground.ipynb) in repository root for detailed
sample output
exceptions are handled by the caller
### `class Behaviour` ### `class Behaviour`
[`typing.NamedTuple`](https://docs.python.org/3/library/typing.html#typing.NamedTuple) [`typing.NamedTuple`](https://docs.python.org/3/library/typing.html#typing.NamedTuple)
@ -629,15 +731,13 @@ attributes
original user-passed query string or a list of strings from splitting user-passed query original user-passed query string or a list of strings from splitting user-passed query
string by spaces string by spaces
- `geocoder: typing.Callable[[str], Latlong] = default_geocoder` - `geocoder: SurplusGeocoderProtocol = default_geocoding.geocoder`
name string to location function, must take in a string and return a name string to location function, see
[`Latlong`](#class-latlong), exceptions are handled by the caller [`SurplusGeocoderProtocol`](#surplusgeocoderprotocol) for more information
- `reverser: Callable[[Latlong], dict[str, Any]] = default_reverser` - `reverser: SurplusReverserProtocol = default_geocoding.reverser`
[`Latlong`](#class-latlong) object to dictionary function, must take in a string and return a Latlong object to address information dictionary function, see
dict. keys found in SHAREABLE_TEXT_LINE_*_KEYS used to access address details are placed [`SurplusReverserProtocol`](#surplusreverserprotocol) for more information
top-level in the dict, exceptions are handled by the caller.
see the [playground notebook](playground.ipynb) for example output
- `stderr: typing.TextIO = sys.stderr` - `stderr: typing.TextIO = sys.stderr`
[TextIO-like object](https://docs.python.org/3/library/io.html#text-i-o) [TextIO-like object](https://docs.python.org/3/library/io.html#text-i-o)
@ -658,6 +758,87 @@ attributes
- `convert_to_type: ConversionResultTypeEnum = ConversionResultTypeEnum.SHAREABLE_TEXT` - `convert_to_type: ConversionResultTypeEnum = ConversionResultTypeEnum.SHAREABLE_TEXT`
what type to convert the query to what type to convert the query to
### `class SurplusDefaultGeocoding`
> [!IMPORTANT]
> this has replaced the now deprecated default geocoding functions, `default_geocoder()`
> and `default_reverser()`, in surplus 2.1.0 and later.
see [SurplusGeocoderProtocol](#surplusgeocoderprotocol) and
[SurplusReverserProtocol](#surplusreverserprotocol) for more information how to
implement a compliant custom geocoder functions.
[`dataclasses.dataclass`](https://docs.python.org/3/library/dataclasses.html) providing
the default geocoding functionality for surplus, via
[OpenStreetMap Nominatim](https://nominatim.openstreetmap.org/)
attributes
- `user_agent: str = default_fingerprint`
pass in a custom user agent here, else it will be the default
[fingerprinted user agent](#details-on-the-fingerprinted-user-agent)
example usage
```python
from surplus import surplus, Behaviour, SurplusDefaultGeocoding
geocoding = SurplusDefaultGeocoding("custom user agent")
geocoding.update_geocoding_functions() # not necessary but recommended
behaviour = Behaviour(
...,
geocoder=geocoding.geocoder,
reverser=geocoding.reverser
)
result = surplus("query", behaviour=behaviour)
...
```
methods
- [`def update_geocoding_functions(self) -> None: ...`](#surplusdefaultgeocodingupdate_geocoding_functions)
- [`def geocoder(self, place: str) -> Latlong: ...`](#surplusdefaultgeocodinggeocoder)
- [`def reverser(self, latlong: Latlong, level: int = 18) -> dict[str, Any]: ...`](#surplusdefaultgeocodingreverser)
#### `SurplusDefaultGeocoding.update_geocoding_functions()`
re-initialise the geocoding functions with the current user agent, also generate a new
user agent if not set properly
it is recommended to call this before using surplus as by default the geocoding functions
are uninitialised
- signature
```python
def update_geocoding_functions(self) -> None: ...
```
#### `SurplusDefaultGeocoding.geocoder()`
> [!WARNING]
> this function is primarily given to be passed into a [`Behaviour`](#class-behaviour)
> object, and is not meant to be called directly.
default geocoder for surplus
see [SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information on surplus
geocoder functions
#### `SurplusDefaultGeocoding.reverser()`
> [!WARNING]
> this function is primarily given to be passed into a [`Behaviour`](#class-behaviour)
> object, and is not meant to be called directly.
default reverser for surplus
see [SurplusReverserProtocol](#surplusreverserprotocol) for more information on surplus
reverser functions
### `class ConversionResultTypeEnum` ### `class ConversionResultTypeEnum`
[enum.Enum](https://docs.python.org/3/library/enum.html) [enum.Enum](https://docs.python.org/3/library/enum.html)
@ -808,15 +989,15 @@ methods
- signature - signature
```python ```python
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]: def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
... ...
``` ```
- arguments - arguments
- `geocoder: typing.Callable[[str], Latlong]` - `geocoder: SurplusGeocoderProtocol`
name string to location function, must take in a string and return a name string to location function, see
[`Latlong`](#class-latlong), exceptions are handled by the caller [SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)[`[Latlong]`](#class-latlong) - returns [`Result`](#class-result)[`[Latlong]`](#class-latlong)
@ -860,15 +1041,15 @@ exclusive method that returns a full-length Plus Code as a string
- signature - signature
```python ```python
def to_full_plus_code(self, geocoder: Callable[[str], Latlong]) -> Result[str]: def to_full_plus_code(self, geocoder: SurplusGeocoderProtocol) -> Result[str]:
... ...
``` ```
- arguments - arguments
- `geocoder: typing.Callable[[str], Latlong]` - `geocoder: SurplusGeocoderProtocol`
name string to location function, must take in a string and return a name string to location function, see
[`Latlong`](#class-latlong), exceptions are handled by the caller [SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)`[str]` - returns [`Result`](#class-result)`[str]`
@ -879,15 +1060,15 @@ method that returns a latitude-longitude coordinate pair
- signature - signature
```python ```python
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]: def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
... ...
``` ```
- arguments - arguments
- `geocoder: typing.Callable[[str], Latlong]` - `geocoder: SurplusGeocoderProtocol`
name string to location function, must take in a string and return a name string to location function, see
[`Latlong`](#class-latlong), exceptions are handled by the caller [SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)[`[Latlong]`](#class-latlong) - returns [`Result`](#class-result)[`[Latlong]`](#class-latlong)
@ -924,15 +1105,15 @@ method that returns a latitude-longitude coordinate pair
- signature - signature
```python ```python
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]: def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
... ...
``` ```
- arguments - arguments
- `geocoder: typing.Callable[[str], Latlong]` - `geocoder: SurplusGeocoderProtocol`
name string to location function, must take in a string and return a name string to location function, see
[`Latlong`](#class-latlong), exceptions are handled by the caller [SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)[`[Latlong]`](#class-latlong) - returns [`Result`](#class-result)[`[Latlong]`](#class-latlong)
@ -969,15 +1150,15 @@ method that returns a latitude-longitude coordinate pair
- signature - signature
```python ```python
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]: def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
... ...
``` ```
- arguments - arguments
- `geocoder: typing.Callable[[str], Latlong]` - `geocoder: SurplusGeocoderProtocol`
name string to location function, must take in a string and return a name string to location function, see
[`Latlong`](#class-latlong), exceptions are handled by the caller [SurplusGeocoderProtocol](#surplusgeocoderprotocol) for more information
- returns [`Result`](#class-result)[`[Latlong]`](#class-latlong) - returns [`Result`](#class-result)[`[Latlong]`](#class-latlong)
@ -1030,36 +1211,91 @@ function that parses a query string into a query object
- returns [`Result`](#class-result)[`[Query]`](#query) - returns [`Result`](#class-result)[`[Query]`](#query)
### `def default_geocoder()` ### `def generate_fingerprinted_user_agent()`
default geocoder for surplus, uses OpenStreetMap Nominatim function that attempts to return a unique user agent string.
> [!NOTE]
> function is not used by surplus and not directly by the user, but is exposed for
> convenience being [Behaviour](#class-behaviour) objects.
> pass in a custom function to [Behaviour](#class-behaviour) to override the default reverser.
- signature - signature
```python ```python
def default_geocoder(place: str) -> Latlong: def generate_fingerprinted_user_agent() -> Result[str]:
```
- returns [`Result[str]`](#class-result)
this result will always have a valid value as erroneous results will have a
resulting value of `'surplus/<version> (generic-user)'`
valid results will have a value of `'surplus/<version> (<fingerprin hasht>)'`, where
the fingerprint hash is a 12 character hexadecimal string
#### details on the fingerprinted user agent
**why do this in the first place?**
if too many people use surplus at the same time,
Nominatim will start to think it's just one person being greedy. so to prevent this,
surplus will try to generate a unique user agent string for each user through
fingerprinting.
at the time of writing, the pre-hashed fingerprint string is as follows:
```python
unique_info: str = f"{version}-{system_info}-{hostname}-{mac_address}"
```
it contains the following, in order, alongside an example:
1. `version` - the surplus version alongside a suffix, if any
```text
2.1.0-local
``` ```
### `def default_reverser()` 2. `system_info` - generic machine and operating system information
default reverser for surplus, uses OpenStreetMap Nominatim ```text
Linux-6.5.0-locietta-WSL2-xanmod1-x86_64-with-glibc2.35
> [!NOTE]
> function is not used by surplus and not directly by the user, but is exposed for
> convenience being [Behaviour](#class-behaviour) objects.
> pass in a custom function to [Behaviour](#class-behaviour) to override the default reverser.
- signature
```python
def default_reverser(latlong: Latlong) -> dict[str, Any]:
``` ```
3. `hostname` - your computer's hostname
```text
mark
```
4. `mac_address` - your computer's mac address
```text
A9:36:3C:98:79:33
```
after hashing, this string becomes a 12 character hexadecimal string, as shown below:
```text
surplus/2.1.0-local (1fdbfa0b0cfb)
^^^^^^^^^^
this is the hashed result of unique_info
```
if at any time, the retrieval of ant of these four elements fail, surplus will just give
up and default to `'surplus/<version> (generic-user)'`.
if any of this seems weird to you, that's fine. pass in a custom user agent flag to
surplus with `-u` or `--user-agent` to override the default user agent, or override the
default user agent in your own code by passing in a custom user agent string to
[`Behaviour`](#class-behaviour).
```text
$ surplus --user_agent "a-shiny-custom-and-unique-user-agent" 77Q4+7X Austin, Texas, USA
...
```
```python
>>> from surplus import surplus, Behaviour
>>> surplus(..., Behaviour(user_agent="a-shiny-custom-and-unique-user-agent"))
...
```
## licence ## licence
surplus is free and unencumbered software released into the public domain. for more surplus is free and unencumbered software released into the public domain. for more
@ -1069,21 +1305,15 @@ python module docstring.
however, direct dependencies of surplus are licensed under different, but still permissive however, direct dependencies of surplus are licensed under different, but still permissive
and open-source licences. and open-source licences.
```text
geopy 2.4.0 Python Geocoding Toolbox
└── geographiclib >=1.52,<3
pluscodes 2022.1.3 Compute Plus Codes (Open Location Codes).
```
- [geopy](https://pypi.org/project/geopy/): - [geopy](https://pypi.org/project/geopy/):
Python Geocoding Toolbox Python Geocoding Toolbox
MIT License MIT Licence
- [geographiclib](https://pypi.org/project/geographiclib/): - [geographiclib](https://pypi.org/project/geographiclib/):
The geodesic routines from GeographicLib The geodesic routines from GeographicLib
MIT License MIT Licence
- [pluscodes](https://pypi.org/project/pluscodes/): - [pluscodes](https://pypi.org/project/pluscodes/):
Compute Plus Codes (Open Location Codes) Compute Plus Codes (Open Location Codes)

288
playground.ipynb
View file

@ -4,7 +4,7 @@
"cell_type": "markdown", "cell_type": "markdown",
"metadata": {}, "metadata": {},
"source": [ "source": [
"# surplus 2.0.0 playground notebook\n", "# surplus 2.x.y playground notebook\n",
"\n", "\n",
"wrangling with environments for devbox users using codium/vs code:\n", "wrangling with environments for devbox users using codium/vs code:\n",
"\n", "\n",
@ -42,13 +42,15 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 1, "execution_count": 2,
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
"source": [ "source": [
"from surplus import PlusCodeQuery, LocalCodeQuery, LatlongQuery, StringQuery\n", "from surplus import PlusCodeQuery, LocalCodeQuery, LatlongQuery, StringQuery\n",
"from surplus import Latlong, Result\n", "from surplus import Latlong, Result\n",
"from surplus import default_geocoder, default_reverser" "from surplus import SurplusDefaultGeocoding\n",
"\n",
"geocoding = SurplusDefaultGeocoding()"
] ]
}, },
{ {
@ -60,7 +62,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 2, "execution_count": 3,
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -78,9 +80,9 @@
"traceback": [ "traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)", "\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m/home/m/works/surplus/playground.ipynb Cell 5\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=7'>8</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\u001b[39mbool\u001b[39m(nom_result), \u001b[39mrepr\u001b[39m(nom_result\u001b[39m.\u001b[39merror), nom_result\u001b[39m.\u001b[39mget()))\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=8'>9</a>\u001b[0m \u001b[39mprint\u001b[39m(\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=9'>10</a>\u001b[0m \u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=10'>11</a>\u001b[0m \u001b[39mbool\u001b[39m(exc_result), \u001b[39mrepr\u001b[39m(exc_result\u001b[39m.\u001b[39merror), exc_result\u001b[39m.\u001b[39mcry(string\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m)\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=11'>12</a>\u001b[0m )\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=12'>13</a>\u001b[0m )\n\u001b[0;32m---> <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=13'>14</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\u001b[39mbool\u001b[39m(exc_result), \u001b[39mrepr\u001b[39m(exc_result\u001b[39m.\u001b[39merror), exc_result\u001b[39m.\u001b[39;49mget()))\n", "\u001b[1;32m/home/m/works/surplus/playground.ipynb Cell 5\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=7'>8</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\u001b[39mbool\u001b[39m(nom_result), \u001b[39mrepr\u001b[39m(nom_result\u001b[39m.\u001b[39merror), nom_result\u001b[39m.\u001b[39mget()))\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=8'>9</a>\u001b[0m \u001b[39mprint\u001b[39m(\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=9'>10</a>\u001b[0m \u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=10'>11</a>\u001b[0m \u001b[39mbool\u001b[39m(exc_result), \u001b[39mrepr\u001b[39m(exc_result\u001b[39m.\u001b[39merror), exc_result\u001b[39m.\u001b[39mcry(string\u001b[39m=\u001b[39m\u001b[39mTrue\u001b[39;00m)\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=11'>12</a>\u001b[0m )\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=12'>13</a>\u001b[0m )\n\u001b[0;32m---> <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=13'>14</a>\u001b[0m \u001b[39mprint\u001b[39m(\u001b[39m\"\u001b[39m\u001b[39m{}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{:<40}\u001b[39;00m\u001b[39m\\t\u001b[39;00m\u001b[39m{}\u001b[39;00m\u001b[39m\"\u001b[39m\u001b[39m.\u001b[39mformat(\u001b[39mbool\u001b[39m(exc_result), \u001b[39mrepr\u001b[39m(exc_result\u001b[39m.\u001b[39merror), exc_result\u001b[39m.\u001b[39;49mget()))\n",
"File \u001b[0;32m~/works/surplus/surplus.py:247\u001b[0m, in \u001b[0;36mResult.get\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 245\u001b[0m \u001b[39m\"\"\"method that returns self.value if Result is non-erroneous else raises error\"\"\"\u001b[39;00m\n\u001b[1;32m 246\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39merror, \u001b[39mBaseException\u001b[39;00m):\n\u001b[0;32m--> 247\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39merror\n\u001b[1;32m 248\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mvalue\n", "File \u001b[0;32m~/works/surplus/surplus/surplus.py:270\u001b[0m, in \u001b[0;36mResult.get\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 268\u001b[0m \u001b[39m\"\"\"method that returns self.value if Result is non-erroneous else raises error\"\"\"\u001b[39;00m\n\u001b[1;32m 269\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39merror, \u001b[39mBaseException\u001b[39;00m):\n\u001b[0;32m--> 270\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39merror\n\u001b[1;32m 271\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mvalue\n",
"\u001b[1;32m/home/m/works/surplus/playground.ipynb Cell 5\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0'>1</a>\u001b[0m nom_result \u001b[39m=\u001b[39m Result[\u001b[39mint\u001b[39m](\u001b[39m3\u001b[39m)\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m----> <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3'>4</a>\u001b[0m \u001b[39m1\u001b[39;49m \u001b[39m/\u001b[39;49m \u001b[39m0\u001b[39;49m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4'>5</a>\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mException\u001b[39;00m \u001b[39mas\u001b[39;00m exc:\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#W4sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5'>6</a>\u001b[0m exc_result \u001b[39m=\u001b[39m Result[\u001b[39mint\u001b[39m](\u001b[39m-\u001b[39m\u001b[39m1\u001b[39m, error\u001b[39m=\u001b[39mexc)\n", "\u001b[1;32m/home/m/works/surplus/playground.ipynb Cell 5\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0'>1</a>\u001b[0m nom_result \u001b[39m=\u001b[39m Result[\u001b[39mint\u001b[39m](\u001b[39m3\u001b[39m)\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m----> <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3'>4</a>\u001b[0m \u001b[39m1\u001b[39;49m \u001b[39m/\u001b[39;49m \u001b[39m0\u001b[39;49m\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=4'>5</a>\u001b[0m \u001b[39mexcept\u001b[39;00m \u001b[39mException\u001b[39;00m \u001b[39mas\u001b[39;00m exc:\n\u001b[1;32m <a href='vscode-notebook-cell://wsl%2Balpine/home/m/works/surplus/playground.ipynb#X26sdnNjb2RlLXJlbW90ZQ%3D%3D?line=5'>6</a>\u001b[0m exc_result \u001b[39m=\u001b[39m Result[\u001b[39mint\u001b[39m](\u001b[39m-\u001b[39m\u001b[39m1\u001b[39m, error\u001b[39m=\u001b[39mexc)\n",
"\u001b[0;31mZeroDivisionError\u001b[0m: division by zero" "\u001b[0;31mZeroDivisionError\u001b[0m: division by zero"
] ]
} }
@ -111,7 +113,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 3, "execution_count": null,
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -120,18 +122,18 @@
"Result(value=Latlong(latitude=1.3336875, longitude=103.7746875), error=None)" "Result(value=Latlong(latitude=1.3336875, longitude=103.7746875), error=None)"
] ]
}, },
"execution_count": 3, "execution_count": 4,
"metadata": {}, "metadata": {},
"output_type": "execute_result" "output_type": "execute_result"
} }
], ],
"source": [ "source": [
"PlusCodeQuery(code=\"6PH58QMF+FV\").to_lat_long_coord(geocoder=default_geocoder)" "PlusCodeQuery(code=\"6PH58QMF+FV\").to_lat_long_coord(geocoder=geocoding.geocoder)"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 4, "execution_count": null,
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -140,22 +142,22 @@
"Result(value=Latlong(latitude=1.3336875, longitude=103.7746875), error=None)" "Result(value=Latlong(latitude=1.3336875, longitude=103.7746875), error=None)"
] ]
}, },
"execution_count": 4, "execution_count": 5,
"metadata": {}, "metadata": {},
"output_type": "execute_result" "output_type": "execute_result"
} }
], ],
"source": [ "source": [
"plus_code = LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_full_plus_code(\n", "plus_code = LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_full_plus_code(\n",
" geocoder=default_geocoder\n", " geocoder=geocoding.geocoder\n",
")\n", ")\n",
"\n", "\n",
"PlusCodeQuery(code=plus_code.get()).to_lat_long_coord(geocoder=default_geocoder)" "PlusCodeQuery(code=plus_code.get()).to_lat_long_coord(geocoder=geocoding.geocoder)"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 5, "execution_count": null,
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -164,20 +166,20 @@
"Result(value=Latlong(latitude=1.3336875, longitude=103.7746875), error=None)" "Result(value=Latlong(latitude=1.3336875, longitude=103.7746875), error=None)"
] ]
}, },
"execution_count": 5, "execution_count": 6,
"metadata": {}, "metadata": {},
"output_type": "execute_result" "output_type": "execute_result"
} }
], ],
"source": [ "source": [
"LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_lat_long_coord(\n", "LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_lat_long_coord(\n",
" geocoder=default_geocoder\n", " geocoder=geocoding.geocoder\n",
")" ")"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 6, "execution_count": null,
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -186,7 +188,7 @@
"Result(value=Latlong(latitude=1.33318835, longitude=103.77461234638255), error=None)" "Result(value=Latlong(latitude=1.33318835, longitude=103.77461234638255), error=None)"
] ]
}, },
"execution_count": 6, "execution_count": 7,
"metadata": {}, "metadata": {},
"output_type": "execute_result" "output_type": "execute_result"
} }
@ -194,12 +196,12 @@
"source": [ "source": [
"LatlongQuery(\n", "LatlongQuery(\n",
" latlong=Latlong(latitude=1.33318835, longitude=103.77461234638255)\n", " latlong=Latlong(latitude=1.33318835, longitude=103.77461234638255)\n",
").to_lat_long_coord(geocoder=default_geocoder)" ").to_lat_long_coord(geocoder=geocoding.geocoder)"
] ]
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 7, "execution_count": null,
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -208,13 +210,13 @@
"Result(value=Latlong(latitude=1.33318835, longitude=103.77461234638255), error=None)" "Result(value=Latlong(latitude=1.33318835, longitude=103.77461234638255), error=None)"
] ]
}, },
"execution_count": 7, "execution_count": 8,
"metadata": {}, "metadata": {},
"output_type": "execute_result" "output_type": "execute_result"
} }
], ],
"source": [ "source": [
"StringQuery(query=\"Ngee Ann Polytechnic\").to_lat_long_coord(geocoder=default_geocoder)" "StringQuery(query=\"Ngee Ann Polytechnic\").to_lat_long_coord(geocoder=geocoding.geocoder)"
] ]
}, },
{ {
@ -282,7 +284,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 7, "execution_count": null,
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [
{ {
@ -298,6 +300,7 @@
" 'house_number': '535',\n", " 'house_number': '535',\n",
" 'latitude': 1.33318835,\n", " 'latitude': 1.33318835,\n",
" 'longitude': 103.77461234638255,\n", " 'longitude': 103.77461234638255,\n",
" 'neighbourhood': 'Ewart Park',\n",
" 'postcode': '599489',\n", " 'postcode': '599489',\n",
" 'raw': {'address': {'ISO3166-2-lvl6': 'SG-03',\n", " 'raw': {'address': {'ISO3166-2-lvl6': 'SG-03',\n",
" 'amenity': 'Ngee Ann Polytechnic',\n", " 'amenity': 'Ngee Ann Polytechnic',\n",
@ -306,6 +309,7 @@
" 'country_code': 'sg',\n", " 'country_code': 'sg',\n",
" 'county': 'Northwest',\n", " 'county': 'Northwest',\n",
" 'house_number': '535',\n", " 'house_number': '535',\n",
" 'neighbourhood': 'Ewart Park',\n",
" 'postcode': '599489',\n", " 'postcode': '599489',\n",
" 'road': 'Clementi Road',\n", " 'road': 'Clementi Road',\n",
" 'suburb': 'Bukit Timah'},\n", " 'suburb': 'Bukit Timah'},\n",
@ -315,8 +319,9 @@
" '103.7701481',\n", " '103.7701481',\n",
" '103.7783945'],\n", " '103.7783945'],\n",
" 'class': 'amenity',\n", " 'class': 'amenity',\n",
" 'display_name': 'Ngee Ann Polytechnic, 535, Clementi Road, Bukit '\n", " 'display_name': 'Ngee Ann Polytechnic, 535, Clementi Road, Ewart '\n",
" 'Timah, Singapore, Northwest, 599489, Singapore',\n", " 'Park, Bukit Timah, Singapore, Northwest, 599489, '\n",
" 'Singapore',\n",
" 'importance': 0.34662169301918117,\n", " 'importance': 0.34662169301918117,\n",
" 'lat': '1.33318835',\n", " 'lat': '1.33318835',\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. '\n", " 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. '\n",
@ -325,7 +330,7 @@
" 'name': 'Ngee Ann Polytechnic',\n", " 'name': 'Ngee Ann Polytechnic',\n",
" 'osm_id': 2535118,\n", " 'osm_id': 2535118,\n",
" 'osm_type': 'relation',\n", " 'osm_type': 'relation',\n",
" 'place_id': 297946059,\n", " 'place_id': 250910125,\n",
" 'place_rank': 30,\n", " 'place_rank': 30,\n",
" 'type': 'university'},\n", " 'type': 'university'},\n",
" 'road': 'Clementi Road',\n", " 'road': 'Clementi Road',\n",
@ -337,15 +342,240 @@
"import pprint\n", "import pprint\n",
"\n", "\n",
"latlong = LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_lat_long_coord(\n", "latlong = LocalCodeQuery(code=\"8QMF+FV\", locality=\"Singapore\").to_lat_long_coord(\n",
" default_geocoder\n", " geocoder=geocoding.geocoder\n",
")\n", ")\n",
"if not latlong:\n", "if not latlong:\n",
" latlong.cry()\n", " latlong.cry()\n",
"\n", "\n",
"else:\n", "else:\n",
" location = default_reverser(latlong.get())\n", " location = geocoding.reverser(latlong.get())\n",
" pprint.pprint(location)" " pprint.pprint(location)"
] ]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2.1.0: the adventure of shortening global/full Plus Codes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### testing rate-limited default geocoding functions"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.33318835, 103.77461234638255\n",
"{'amenity': 'Ngee Ann Polytechnic', 'house_number': '535', 'road': 'Clementi Road', 'neighbourhood': 'Ewart Park', 'suburb': 'Bukit Timah', 'city': 'Singapore', 'county': 'Northwest', 'ISO3166-2-lvl6': 'SG-03', 'postcode': '599489', 'country': 'Singapore', 'country_code': 'sg', 'raw': {'place_id': 250910125, 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright', 'osm_type': 'relation', 'osm_id': 2535118, 'lat': '1.33318835', 'lon': '103.77461234638255', 'class': 'amenity', 'type': 'university', 'place_rank': 30, 'importance': 0.34662169301918117, 'addresstype': 'amenity', 'name': 'Ngee Ann Polytechnic', 'display_name': 'Ngee Ann Polytechnic, 535, Clementi Road, Ewart Park, Bukit Timah, Singapore, Northwest, 599489, Singapore', 'address': {'amenity': 'Ngee Ann Polytechnic', 'house_number': '535', 'road': 'Clementi Road', 'neighbourhood': 'Ewart Park', 'suburb': 'Bukit Timah', 'city': 'Singapore', 'county': 'Northwest', 'ISO3166-2-lvl6': 'SG-03', 'postcode': '599489', 'country': 'Singapore', 'country_code': 'sg'}, 'boundingbox': ['1.3289692', '1.3372184', '103.7701481', '103.7783945']}, 'latitude': 1.33318835, 'longitude': 103.77461234638255}\n"
]
}
],
"source": [
"from surplus import SurplusGeocoderProtocol, SurplusReverserProtocol\n",
"\n",
"\n",
"test_geocoding = SurplusDefaultGeocoding(user_agent=\"surplus/playground\")\n",
"\n",
"print(location := test_geocoding.geocoder(\"Ngee Ann Polytechnic\"))\n",
"\n",
"print(reversed := test_geocoding.reverser(f\"{location.latitude}, {location.longitude}\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### loop for less information until a local code is made"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO\n",
"\n",
"test1 = LocalCodeQuery(\"9R3J+R9\", \"Singapore\")\n",
"test2 = LocalCodeQuery(\"G227+XF\", \"St Lucia, Queensland, Australia\")\n",
"\n",
"level = 13"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'suburb': 'Bishan',\n",
" 'city': 'Singapore',\n",
" 'county': 'Central',\n",
" 'ISO3166-2-lvl6': 'SG-01',\n",
" 'country': 'Singapore',\n",
" 'country_code': 'sg',\n",
" 'raw': {'place_id': 251115282,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
" 'osm_type': 'way',\n",
" 'osm_id': 795946716,\n",
" 'lat': '1.3519117',\n",
" 'lon': '103.8489708',\n",
" 'class': 'place',\n",
" 'type': 'suburb',\n",
" 'place_rank': 19,\n",
" 'importance': 0.39184907371668787,\n",
" 'addresstype': 'suburb',\n",
" 'name': 'Bishan',\n",
" 'display_name': 'Bishan, Singapore, Central, Singapore',\n",
" 'address': {'suburb': 'Bishan',\n",
" 'city': 'Singapore',\n",
" 'county': 'Central',\n",
" 'ISO3166-2-lvl6': 'SG-01',\n",
" 'country': 'Singapore',\n",
" 'country_code': 'sg'},\n",
" 'boundingbox': ['1.3416846', '1.3679829', '103.8184512', '103.8604083']},\n",
" 'latitude': 1.3519117,\n",
" 'longitude': 103.8489708}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(\n",
" response := geocoding.reverser(\n",
" test1.to_lat_long_coord(geocoding.geocoder).get(), level=level\n",
" )\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'suburb': 'St Lucia',\n",
" 'city_district': 'St Lucia',\n",
" 'city': 'Brisbane City',\n",
" 'state': 'Queensland',\n",
" 'ISO3166-2-lvl4': 'AU-QLD',\n",
" 'postcode': '4072',\n",
" 'country': 'Australia',\n",
" 'country_code': 'au',\n",
" 'raw': {'place_id': 54477898,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
" 'osm_type': 'node',\n",
" 'osm_id': 88800268,\n",
" 'lat': '-27.4987362',\n",
" 'lon': '153.0081642',\n",
" 'class': 'place',\n",
" 'type': 'suburb',\n",
" 'place_rank': 19,\n",
" 'importance': 0.27501,\n",
" 'addresstype': 'suburb',\n",
" 'name': 'St Lucia',\n",
" 'display_name': 'St Lucia, Brisbane City, Queensland, 4072, Australia',\n",
" 'address': {'suburb': 'St Lucia',\n",
" 'city_district': 'St Lucia',\n",
" 'city': 'Brisbane City',\n",
" 'state': 'Queensland',\n",
" 'ISO3166-2-lvl4': 'AU-QLD',\n",
" 'postcode': '4072',\n",
" 'country': 'Australia',\n",
" 'country_code': 'au'},\n",
" 'boundingbox': ['-27.5187362', '-27.4787362', '152.9881642', '153.0281642']},\n",
" 'latitude': -27.4987362,\n",
" 'longitude': 153.0081642}"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"(\n",
" response := geocoding.reverser(\n",
" test2.to_lat_long_coord(geocoding.geocoder).get(), level=level\n",
" )\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## machine fingerprinting attempt\n",
"\n",
"because of nominatim's acceptable usage policy \n",
"<https://operations.osmfoundation.org/policies/nominatim/>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from hashlib import shake_256 as _hashlib_shake_256\n",
"from platform import platform as _platform_platform\n",
"from socket import gethostname as _socket_gethostname\n",
"from uuid import getnode as _uuid_getnode\n",
"from surplus import VERSION, VERSION_SUFFIX\n",
"\n",
"\n",
"def generate_fingerprinted_user_agent() -> Result[str]:\n",
" \"\"\"\n",
" function that attempts to return a unique user agent string.\n",
"\n",
" returns Result[str]\n",
" this result will always have a valid value as erroneous results will have a\n",
" resulting value of 'surplus/<version>/generic-user'\n",
" valid results will have a value of 'surplus/<version>/<fingerprint>', where\n",
" fingerprint is a 12 character hexadecimal string\n",
" \"\"\"\n",
" version: str = \".\".join([str(v) for v in VERSION]) + VERSION_SUFFIX\n",
"\n",
" try:\n",
" system_info: str = _platform_platform()\n",
" hostname: str = _socket_gethostname()\n",
" mac_address: str = \":\".join(\n",
" [\n",
" \"{:02x}\".format((_uuid_getnode() >> elements) & 0xFF)\n",
" for elements in range(0, 2 * 6, 2)\n",
" ][::-1]\n",
" )\n",
" unique_info: str = f\"{version}-{system_info}-{hostname}-{mac_address}\"\n",
"\n",
" print(f\"{version=}\")\n",
" print(f\"{system_info=}\")\n",
" print(f\"{hostname=}\")\n",
" print(f\"{mac_address=}\")\n",
"\n",
" except Exception as exc:\n",
" return Result[str](f\"surplus/{version} (generic-user)\", error=exc)\n",
"\n",
" fingerprint: str = _hashlib_shake_256(unique_info.encode()).hexdigest(5)\n",
"\n",
" return Result[str](f\"surplus/{version} ({fingerprint})\")"
]
} }
], ],
"metadata": { "metadata": {

6
poetry.lock generated
View file

@ -14,14 +14,14 @@ files = [
[[package]] [[package]]
name = "asttokens" name = "asttokens"
version = "2.3.0" version = "2.4.0"
description = "Annotate AST trees with source code positions" description = "Annotate AST trees with source code positions"
category = "dev" category = "dev"
optional = false optional = false
python-versions = "*" python-versions = "*"
files = [ files = [
{file = "asttokens-2.3.0-py2.py3-none-any.whl", hash = "sha256:bef1a51bc256d349e9f94e7e40e44b705ed1162f55294220dd561d24583d9877"}, {file = "asttokens-2.4.0-py2.py3-none-any.whl", hash = "sha256:cf8fc9e61a86461aa9fb161a14a0841a03c405fa829ac6b202670b3495d2ce69"},
{file = "asttokens-2.3.0.tar.gz", hash = "sha256:2552a88626aaa7f0f299f871479fc755bd4e7c11e89078965e928fb7bb9a6afe"}, {file = "asttokens-2.4.0.tar.gz", hash = "sha256:2e0171b991b2c959acc6c49318049236844a5da1d65ba2672c4880c1c894834e"},
] ]
[package.dependencies] [package.dependencies]

13
surplus/__init__.py
View file

@ -32,10 +32,14 @@ For more information, please refer to <http://unlicense.org/>
# surplus was and would've been a single-file module, but typing is in the way :( # surplus was and would've been a single-file module, but typing is in the way :(
# https://github.com/python/typing/issues/1333 # https://github.com/python/typing/issues/1333
from .surplus import default_geocoder # deprecated, emulation function
from .surplus import default_reverser # deprecated, emulation function
from .surplus import ( from .surplus import (
BUILD_BRANCH, BUILD_BRANCH,
BUILD_COMMIT, BUILD_COMMIT,
BUILD_DATETIME, BUILD_DATETIME,
CONNECTION_MAX_RETRIES,
CONNECTION_WAIT_SECONDS,
EMPTY_LATLONG, EMPTY_LATLONG,
SHAREABLE_TEXT_LINE_0_KEYS, SHAREABLE_TEXT_LINE_0_KEYS,
SHAREABLE_TEXT_LINE_1_KEYS, SHAREABLE_TEXT_LINE_1_KEYS,
@ -45,7 +49,6 @@ from .surplus import (
SHAREABLE_TEXT_LINE_5_KEYS, SHAREABLE_TEXT_LINE_5_KEYS,
SHAREABLE_TEXT_LINE_6_KEYS, SHAREABLE_TEXT_LINE_6_KEYS,
SHAREABLE_TEXT_NAMES, SHAREABLE_TEXT_NAMES,
USER_AGENT,
VERSION, VERSION,
VERSION_SUFFIX, VERSION_SUFFIX,
Behaviour, Behaviour,
@ -60,13 +63,15 @@ from .surplus import (
PlusCodeNotFoundError, PlusCodeNotFoundError,
PlusCodeQuery, PlusCodeQuery,
Query, Query,
Result,
ResultType, ResultType,
StringQuery, StringQuery,
SurplusDefaultGeocoding,
SurplusException, SurplusException,
UnavailableFeatureError, SurplusGeocoderProtocol,
SurplusReverserProtocol,
cli, cli,
default_geocoder, generate_fingerprinted_user_agent,
default_reverser,
handle_args, handle_args,
parse_query, parse_query,
surplus, surplus,

383
surplus/surplus.py
View file

@ -31,8 +31,13 @@ For more information, please refer to <http://unlicense.org/>
from argparse import ArgumentParser from argparse import ArgumentParser
from collections import OrderedDict from collections import OrderedDict
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone from datetime import datetime, timedelta, timezone
from enum import Enum from enum import Enum
from functools import lru_cache
from hashlib import shake_256
from platform import platform
from socket import gethostname
from sys import stderr, stdin, stdout from sys import stderr, stdin, stdout
from typing import ( from typing import (
Any, Any,
@ -45,12 +50,17 @@ from typing import (
TypeAlias, TypeAlias,
TypeVar, TypeVar,
) )
from uuid import getnode
from geopy import Location as _geopy_Location # type: ignore from geopy import Location as _geopy_Location # type: ignore
from geopy.extra.rate_limiter import RateLimiter as _geopy_RateLimiter # type: ignore
from geopy.geocoders import Nominatim as _geopy_Nominatim # type: ignore from geopy.geocoders import Nominatim as _geopy_Nominatim # type: ignore
from pluscodes import Area as _PlusCode_Area # type: ignore
from pluscodes import PlusCode as _PlusCode # type: ignore from pluscodes import PlusCode as _PlusCode # type: ignore
from pluscodes import decode as _PlusCode_decode # type: ignore
from pluscodes import encode as _PlusCode_encode # type: ignore from pluscodes import encode as _PlusCode_encode # type: ignore
from pluscodes.validator import Validator as _PlusCode_Validator # type: ignore from pluscodes.validator import Validator as _PlusCode_Validator # type: ignore
from typing_extensions import Protocol
from pluscodes.openlocationcode import ( # type: ignore # isort: skip from pluscodes.openlocationcode import ( # type: ignore # isort: skip
recoverNearest as _PlusCode_recoverNearest, recoverNearest as _PlusCode_recoverNearest,
@ -63,7 +73,8 @@ VERSION_SUFFIX: Final[str] = "-local"
BUILD_BRANCH: Final[str] = "future" BUILD_BRANCH: Final[str] = "future"
BUILD_COMMIT: Final[str] = "latest" BUILD_COMMIT: Final[str] = "latest"
BUILD_DATETIME: Final[datetime] = datetime.now(timezone(timedelta(hours=8))) # using SGT BUILD_DATETIME: Final[datetime] = datetime.now(timezone(timedelta(hours=8))) # using SGT
USER_AGENT: Final[str] = "surplus" CONNECTION_MAX_RETRIES: int = 9
CONNECTION_WAIT_SECONDS: int = 10
SHAREABLE_TEXT_LINE_0_KEYS: Final[tuple[str, ...]] = ( SHAREABLE_TEXT_LINE_0_KEYS: Final[tuple[str, ...]] = (
"emergency", "emergency",
"historic", "historic",
@ -155,10 +166,6 @@ class EmptyQueryError(SurplusException):
... ...
class UnavailableFeatureError(SurplusException):
...
# data structures # data structures
@ -290,6 +297,66 @@ class Latlong(NamedTuple):
EMPTY_LATLONG: Final[Latlong] = Latlong(latitude=0.0, longitude=0.0) EMPTY_LATLONG: Final[Latlong] = Latlong(latitude=0.0, longitude=0.0)
class SurplusGeocoderProtocol(Protocol):
"""
typing_extensions.Protocol class for documentation and static type checking of
surplus reverser functions
(place: str) -> Latlong
name string to location function. must take in a string and return a Latlong.
function can be functools.lru_cache()-wrapped if the geocoding service asks for
caching
exceptions are handled by the caller
"""
def __call__(self, place: str) -> Latlong:
...
class SurplusReverserProtocol(Protocol):
"""
typing_extensions.Protocol class for documentation and static type checking of
surplus reverser functions
(latlong: Latlong, level: int = 18) -> dict[str, Any]:
Latlong object to address information dictionary function. must take in a string and
return a dict with SHAREABLE_TEXT_LINE_*_KEYS keys at the dictionaries' top-level.
keys are used to access address information.
function should also take in a int representing the level of detail for the
returned address, 0-18 (country-level to building), inclusive.
keys for latitude, longitude and an iso3166-2 (or closest equivalent) should also be
included at the dictionaries top level as the keys `latitude`, `longitude` and
`ISO3166-2` (non-case sensitive, or at least something starting with `ISO3166`)
respectively.
{
'ISO3166-2-lvl6': 'SG-03',
'amenity': 'Ngee Ann Polytechnic',
...
'country': 'Singapore',
'latitude': 1.33318835,
'longitude': 103.77461234638255,
'postcode': '599489',
'raw': {...},
}
function can be functools.lru_cache()-wrapped if the geocoding service asks for
caching
exceptions are handled by the caller,
see the playground notebook in repository root for sample output
"""
def __call__(self, latlong: Latlong, level: int = 18) -> dict[str, Any]:
...
class PlusCodeQuery(NamedTuple): class PlusCodeQuery(NamedTuple):
""" """
typing.NamedTuple representing a full-length Plus Code (e.g., 6PH58QMF+FX) typing.NamedTuple representing a full-length Plus Code (e.g., 6PH58QMF+FX)
@ -304,14 +371,14 @@ class PlusCodeQuery(NamedTuple):
code: str code: str
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]: def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
""" """
method that returns a latitude-longitude coordinate pair method that returns a latitude-longitude coordinate pair
arguments arguments
geocoder: typing.Callable[[str], Latlong] geocoder: SurplusGeocoderProtocol
name string to location function, must take in a string and return a name string to location function, see SurplusGeocoderProtocol docstring
Latlong, exceptions are handled by the caller for more information
returns Result[Latlong] returns Result[Latlong]
""" """
@ -363,14 +430,14 @@ class LocalCodeQuery(NamedTuple):
code: str code: str
locality: str locality: str
def to_full_plus_code(self, geocoder: Callable[[str], Latlong]) -> Result[str]: def to_full_plus_code(self, geocoder: SurplusGeocoderProtocol) -> Result[str]:
""" """
exclusive method that returns a full-length Plus Code as a string exclusive method that returns a full-length Plus Code as a string
arguments arguments
geocoder: typing.Callable[[str], Latlong] geocoder: SurplusGeocoderProtocol
name string to location function, must take in a string and return a name string to location function, see SurplusGeocoderProtocol docstring
Latlong, exceptions are handled by the caller for more information
returns Result[str] returns Result[str]
""" """
@ -389,14 +456,14 @@ class LocalCodeQuery(NamedTuple):
except Exception as exc: except Exception as exc:
return Result[str]("", error=exc) return Result[str]("", error=exc)
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]: def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
""" """
method that returns a latitude-longitude coordinate pair method that returns a latitude-longitude coordinate pair
arguments arguments
geocoder: typing.Callable[[str], Latlong] geocoder: SurplusGeocoderProtocol
name string to location function, must take in a string and return a name string to location function, see SurplusGeocoderProtocol docstring
Latlong, exceptions are handled by the caller for more information
returns Result[Latlong] returns Result[Latlong]
""" """
@ -431,14 +498,14 @@ class LatlongQuery(NamedTuple):
latlong: Latlong latlong: Latlong
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]: def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
""" """
method that returns a latitude-longitude coordinate pair method that returns a latitude-longitude coordinate pair
arguments arguments
geocoder: typing.Callable[[str], Latlong] geocoder: SurplusGeocoderProtocol
name string to location function, must take in a string and return a name string to location function, see SurplusGeocoderProtocol docstring
Latlong, exceptions are handled by the caller for more information
returns Result[Latlong] returns Result[Latlong]
""" """
@ -464,14 +531,14 @@ class StringQuery(NamedTuple):
query: str query: str
def to_lat_long_coord(self, geocoder: Callable[[str], Latlong]) -> Result[Latlong]: def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
""" """
method that returns a latitude-longitude coordinate pair method that returns a latitude-longitude coordinate pair
arguments arguments
geocoder: typing.Callable[[str], Latlong] geocoder: SurplusGeocoderProtocol
name string to location function, must take in a string and return a name string to location function, see SurplusGeocoderProtocol docstring
Latlong, exceptions are handled by the caller for more information
returns Result[Latlong] returns Result[Latlong]
""" """
@ -490,12 +557,116 @@ class StringQuery(NamedTuple):
Query: TypeAlias = PlusCodeQuery | LocalCodeQuery | LatlongQuery | StringQuery Query: TypeAlias = PlusCodeQuery | LocalCodeQuery | LatlongQuery | StringQuery
def default_geocoder(place: str) -> Latlong: def generate_fingerprinted_user_agent() -> Result[str]:
"""default geocoder for surplus, uses OpenStreetMap Nominatim""" """
function that attempts to return a unique user agent string.
location: _geopy_Location | None = _geopy_Nominatim(user_agent=USER_AGENT).geocode( returns Result[str]
place this result will always have a valid value as erroneous results will have a
resulting value of 'surplus/<version> (generic-user)'
valid results will have a value of 'surplus/<version> (<fingerprint hash>)',
where <fingerprint hash> is a 12 character hexadecimal string
"""
version: str = ".".join([str(v) for v in VERSION]) + VERSION_SUFFIX
try:
system_info: str = platform()
hostname: str = gethostname()
mac_address: str = ":".join(
[
"{:02x}".format((getnode() >> elements) & 0xFF)
for elements in range(0, 2 * 6, 2)
][::-1]
) )
unique_info: str = f"{version}-{system_info}-{hostname}-{mac_address}"
except Exception as exc:
return Result[str](f"surplus/{version} (generic-user)", error=exc)
fingerprint: str = shake_256(unique_info.encode()).hexdigest(5)
return Result[str](f"surplus/{version} ({fingerprint})")
default_fingerprint: Final[str] = generate_fingerprinted_user_agent().value
@dataclass
class SurplusDefaultGeocoding:
"""
dataclass providing the default geocoding functionality for surplus, via
OpenStreetMap Nominatim
attributes
user_agent: str = default_fingerprint
pass in a custom user agent here, else it will be the default fingerprinted
user agent
usage
geocoding = SurplusDefaultGeocoding(behaviour.user_agent)
geocoding.update_geocoding_functions()
...
Behaviour(
...,
geocoder=geocoding.geocoder,
reverser=geocoding.reverser
)
"""
user_agent: str = default_fingerprint
_ratelimited_raw_geocoder: Callable | None = None
_ratelimited_raw_reverser: Callable | None = None
_first_update: bool = False
def update_geocoding_functions(self) -> None:
"""
re-initialise the geocoding functions with the current user agent, also generate
a new user agent if not set properly
recommended to call this before using surplus as by default the geocoding
functions are uninitialised
"""
if not isinstance(self.user_agent, str):
self.user_agent: str = generate_fingerprinted_user_agent().value
nominatim = _geopy_Nominatim(user_agent=self.user_agent)
# this is
self._ratelimited_raw_geocoder: Callable = lru_cache(
_geopy_RateLimiter(
nominatim.geocode,
max_retries=CONNECTION_MAX_RETRIES,
error_wait_seconds=CONNECTION_WAIT_SECONDS,
)
)
self._ratelimited_raw_reverser: Callable = lru_cache(
_geopy_RateLimiter(
nominatim.reverse,
max_retries=CONNECTION_MAX_RETRIES,
error_wait_seconds=CONNECTION_WAIT_SECONDS,
)
)
self._first_update = True
def geocoder(self, place: str) -> Latlong:
"""
default geocoder for surplus, uses OpenStreetMap Nominatim
see SurplusGeocoderProtocol for more information on surplus geocoder functions
"""
if not callable(self._ratelimited_raw_geocoder) or (self._first_update is False):
self.update_geocoding_functions()
# https://github.com/python/mypy/issues/12155
assert callable(self._ratelimited_raw_geocoder)
location: _geopy_Location | None = self._ratelimited_raw_geocoder(place)
if location is None: if location is None:
raise NoSuitableLocationError( raise NoSuitableLocationError(
@ -507,11 +678,26 @@ def default_geocoder(place: str) -> Latlong:
longitude=location.longitude, longitude=location.longitude,
) )
def reverser(self, latlong: Latlong, level: int = 18) -> dict[str, Any]:
"""
default reverser for surplus, uses OpenStreetMap Nominatim
def default_reverser(latlong: Latlong) -> dict[str, Any]: arguments
"""default reverser for surplus, uses OpenStreetMap Nominatim""" latlong: Latlong
location: _geopy_Location | None = _geopy_Nominatim(user_agent=USER_AGENT).reverse( level: int = 0
str(latlong) level of detail for the returned address, 0-18 (country-building) inclusive
see SurplusReverserProtocol for more information on surplus reverser functions
"""
if not callable(self._ratelimited_raw_reverser) or (self._first_update is False):
self.update_geocoding_functions()
# https://github.com/python/mypy/issues/12155
assert callable(self._ratelimited_raw_reverser)
location: _geopy_Location | None = self._ratelimited_raw_reverser(
str(latlong), zoom=level
) )
if location is None: if location is None:
@ -529,6 +715,34 @@ def default_reverser(latlong: Latlong) -> dict[str, Any]:
return location_dict return location_dict
default_geocoding: Final[SurplusDefaultGeocoding] = SurplusDefaultGeocoding(
default_fingerprint
)
default_geocoding.update_geocoding_functions()
def default_geocoder(place: str) -> Latlong:
"""(deprecated) geocoder for surplus, uses OpenStreetMap Nominatim"""
print(
"warning: default_geocoder is deprecated. "
"this is a emulation function that will use a fingerprinted user agent.",
file=stderr,
)
return default_geocoding.geocoder(place=place)
def default_reverser(latlong: Latlong, level: int = 18) -> dict[str, Any]:
"""
(deprecated) reverser for surplus, uses OpenStreetMap Nominatim
"""
print(
"warning: default_reverser is deprecated. "
"this is a emulation function that will use a fingerprinted user agent.",
file=stderr,
)
return default_geocoding.reverser(latlong=latlong, level=level)
class Behaviour(NamedTuple): class Behaviour(NamedTuple):
""" """
typing.NamedTuple representing how surplus operations should behave typing.NamedTuple representing how surplus operations should behave
@ -537,14 +751,12 @@ class Behaviour(NamedTuple):
query: str | list[str] = "" query: str | list[str] = ""
original user-passed query string or a list of strings from splitting original user-passed query string or a list of strings from splitting
user-passed query string by spaces user-passed query string by spaces
geocoder: Callable[[str], Latlong] = default_geocoderi geocoder: SurplusGeocoderProtocol = default_geocoding.geocoder
name string to location function, must take in a string and return a Latlong, name string to location function, see SurplusGeocoderProtocol docstring for
exceptions are handled by the caller for more information
reverser: Callable[[str], dict[str, Any]] = default_reverser reverser: SurplusReverserProtocol = default_geocoding.reverser
Latlong object to dictionary function, must take in a string and return a latlong to address information dict function, see SurplusReverserProtocol
dict. keys found in SHAREABLE_TEXT_LINE_*_KEYS used to access address details docstring for more information
are placed top-level in the dict, exceptions are handled by the caller.
see the playground notebook for example output
stderr: TextIO = sys.stderr stderr: TextIO = sys.stderr
TextIO-like object representing a writeable file. defaults to sys.stderr TextIO-like object representing a writeable file. defaults to sys.stderr
stdout: TextIO = sys.stdout stdout: TextIO = sys.stdout
@ -558,8 +770,8 @@ class Behaviour(NamedTuple):
""" """
query: str | list[str] = "" query: str | list[str] = ""
geocoder: Callable[[str], Latlong] = default_geocoder geocoder: SurplusGeocoderProtocol = default_geocoding.geocoder
reverser: Callable[[Latlong], dict[str, Any]] = default_reverser reverser: SurplusReverserProtocol = default_geocoding.reverser
stderr: TextIO = stderr stderr: TextIO = stderr
stdout: TextIO = stdout stdout: TextIO = stdout
debug: bool = False debug: bool = False
@ -694,7 +906,7 @@ def parse_query(behaviour: Behaviour) -> Result[Query]:
split_query = behaviour.query split_query = behaviour.query
if behaviour.debug: if behaviour.debug:
print(f"debug: {split_query=}\ndebug: {original_query=}", behaviour.stderr) print(f"debug: {split_query=}\ndebug: {original_query=}", file=behaviour.stderr)
# not a plus/local code, try to match for latlong or string query # not a plus/local code, try to match for latlong or string query
match split_query: match split_query:
@ -804,6 +1016,13 @@ def handle_args() -> Behaviour:
f"'{Behaviour([]).convert_to_type.value}'" f"'{Behaviour([]).convert_to_type.value}'"
), ),
default=Behaviour([]).convert_to_type.value, default=Behaviour([]).convert_to_type.value,
),
parser.add_argument(
"-u",
"--user-agent",
type=str,
help=f"user agent string to use for geocoding service, defaults to fingerprinted user agent string",
default=default_fingerprint,
) )
args = parser.parse_args() args = parser.parse_args()
@ -821,10 +1040,12 @@ def handle_args() -> Behaviour:
else: else:
query = args.query query = args.query
geocoding = SurplusDefaultGeocoding(args.user_agent)
behaviour = Behaviour( behaviour = Behaviour(
query=query, query=query,
geocoder=default_geocoder, geocoder=geocoding.geocoder,
reverser=default_reverser, reverser=geocoding.reverser,
stderr=stderr, stderr=stderr,
stdout=stdout, stdout=stdout,
debug=args.debug, debug=args.debug,
@ -911,7 +1132,7 @@ def _generate_text(
# get iso3166-2 before doing anything # get iso3166-2 before doing anything
iso3166_2: str = "" iso3166_2: str = ""
for key in location: for key in location:
if key.startswith("iso3166"): if key.lower().startswith("iso3166"):
iso3166_2 = location.get(key, "") iso3166_2 = location.get(key, "")
# skeleton code to allow for changing keys based on iso3166-2 code # skeleton code to allow for changing keys based on iso3166-2 code
@ -1004,17 +1225,19 @@ def surplus(query: Query | str, behaviour: Behaviour) -> Result[str]:
match behaviour.convert_to_type: match behaviour.convert_to_type:
case ConversionResultTypeEnum.SHAREABLE_TEXT: case ConversionResultTypeEnum.SHAREABLE_TEXT:
# get latlong and handle result # get latlong and handle result
latlong = query.to_lat_long_coord(geocoder=behaviour.geocoder) latlong_result: Result[Latlong] = query.to_lat_long_coord(
geocoder=behaviour.geocoder
)
if not latlong: if not latlong_result:
return Result[str]("", error=latlong.error) return Result[str]("", error=latlong_result.error)
if behaviour.debug: if behaviour.debug:
print(f"debug: cli: {latlong.get()=}", file=behaviour.stderr) print(f"debug: cli: {latlong_result.get()=}", file=behaviour.stderr)
# reverse location and handle result # reverse location and handle result
try: try:
location: dict[str, Any] = behaviour.reverser(latlong.get()) location: dict[str, Any] = behaviour.reverser(latlong_result.get())
except Exception as exc: except Exception as exc:
return Result[str]("", error=exc) return Result[str]("", error=exc)
@ -1041,21 +1264,23 @@ def surplus(query: Query | str, behaviour: Behaviour) -> Result[str]:
return Result[str](text) return Result[str](text)
case ConversionResultTypeEnum.PLUS_CODE: case ConversionResultTypeEnum.PLUS_CODE:
# if its already a plus code, just return it
if isinstance(query, PlusCodeQuery): if isinstance(query, PlusCodeQuery):
return Result[str](str(query)) return Result[str](str(query))
# get latlong and handle result # get latlong and handle result
latlong = query.to_lat_long_coord(geocoder=behaviour.geocoder) latlong_query = query.to_lat_long_coord(geocoder=behaviour.geocoder)
if not latlong: if not latlong_query:
return Result[str]("", error=latlong.error) return Result[str]("", error=latlong_query.error)
if behaviour.debug: if behaviour.debug:
print(f"debug: cli: {latlong.get()=}", file=behaviour.stderr) print(f"debug: cli: {latlong_query.get()=}", file=behaviour.stderr)
# perform operation
try: try:
pluscode: str = _PlusCode_encode( pluscode: str = _PlusCode_encode(
lat=latlong.get().latitude, lon=latlong.get().longitude lat=latlong_query.get().latitude, lon=latlong_query.get().longitude
) )
except Exception as exc: except Exception as exc:
@ -1064,10 +1289,41 @@ def surplus(query: Query | str, behaviour: Behaviour) -> Result[str]:
return Result[str](pluscode) return Result[str](pluscode)
case ConversionResultTypeEnum.LOCAL_CODE: case ConversionResultTypeEnum.LOCAL_CODE:
# if its already a local code, just return it
if isinstance(query, LocalCodeQuery):
return Result[str](str(query))
latlong: Latlong = EMPTY_LATLONG
# if its a plus code, convert to latlong first
if isinstance(query, PlusCodeQuery):
pluscode_latlong_result = PlusCodeQuery.to_lat_long_coord(
query, geocoder=behaviour.geocoder
)
if not pluscode_latlong_result:
return Result[str]("", error=pluscode_latlong_result.error)
latlong = pluscode_latlong_result.get()
# get latlong and handle result
latlong_result = query.to_lat_long_coord(geocoder=behaviour.geocoder)
if not latlong_result:
return Result[str]("", error=latlong_result.error)
if behaviour.debug:
print(f"debug: cli: {latlong_result.get()=}", file=behaviour.stderr)
latlong = latlong_result.get()
# perform operation
# TODO: https://github.com/markjoshwel/surplus/issues/18 # TODO: https://github.com/markjoshwel/surplus/issues/18
# https://github.com/google/open-location-code/wiki/Guidance-for-shortening-codes
return Result[str]( return Result[str](
text, text,
error=UnavailableFeatureError( error=NotImplementedError(
"converting to Plus Code is not implemented yet" "converting to Plus Code is not implemented yet"
), ),
) )
@ -1078,15 +1334,16 @@ def surplus(query: Query | str, behaviour: Behaviour) -> Result[str]:
return Result[str](str(query)) return Result[str](str(query))
# get latlong and handle result # get latlong and handle result
latlong = query.to_lat_long_coord(geocoder=behaviour.geocoder) latlong_result = query.to_lat_long_coord(geocoder=behaviour.geocoder)
if not latlong: if not latlong_result:
return Result[str]("", error=latlong.error) return Result[str]("", error=latlong_result.error)
if behaviour.debug: if behaviour.debug:
print(f"debug: cli: {latlong.get()=}", file=behaviour.stderr) print(f"debug: cli: {latlong_result.get()=}", file=behaviour.stderr)
return Result[str](str(latlong.get())) # perform operation
return Result[str](str(latlong_result.get()))
case _: case _:
return Result[str]( return Result[str](

12
test.py
View file

@ -100,22 +100,26 @@ tests: list[ContinuityTest] = [
), ),
ContinuityTest( ContinuityTest(
query="Ngee Ann Polytechnic, Singapore", query="Ngee Ann Polytechnic, Singapore",
expected=( expected=[
(
"Ngee Ann Polytechnic\n" "Ngee Ann Polytechnic\n"
"535 Clementi Road\n" "535 Clementi Road\n"
"Bukit Timah\n" "Bukit Timah\n"
"599489\n" "599489\n"
"Northwest, Singapore" "Northwest, Singapore"
), )
],
), ),
ContinuityTest( ContinuityTest(
query="1.3521, 103.8198", query="1.3521, 103.8198",
expected=( expected=[
(
"MacRitchie Nature Trail\n" "MacRitchie Nature Trail\n"
"Central Water Catchment\n" "Central Water Catchment\n"
"574325\n" "574325\n"
"Central, Singapore" "Central, Singapore"
), )
],
), ),
] ]