2023-06-02 19:39:25 +00:00
# surplus
2023-08-31 14:19:14 +00:00
surplus is a Python script to convert
2023-09-01 13:38:03 +00:00
[Google Maps Plus Codes ](https://maps.google.com/pluscodes/ )
to iOS Shortcuts-like shareable text.
2023-08-31 14:19:14 +00:00
2023-09-04 14:37:05 +00:00
> [!NOTE]
> you are on the `future` branch.
> this branch contains the latest development changes to surplus, but may be unstable.
>
> checkout the [`main`](https://github.com/markjoshwel/surplus/tree/main) branch for the
> a stable version of the repository.
2023-08-31 14:19:14 +00:00
- [installation ](#installation )
2023-09-02 20:05:37 +00:00
- [usage ](#usage )
2023-09-03 14:23:40 +00:00
- [command-line usage ](#command-line-usage )
2023-09-02 20:05:37 +00:00
- [example api usage ](#example-api-usage )
2023-08-31 14:19:14 +00:00
- [developer's guide ](#developers-guide )
- [contributor's guide ](#contributors-guide )
- [reporting incorrect output ](#reporting-incorrect-output )
- [the reporting process ](#the-reporting-process )
- [what counts as "incorrect" ](#what-counts-as-incorrect )
- [output technical details ](#the-technical-details-of-surpluss-output )
2023-09-02 20:05:37 +00:00
- [api reference ](#api-reference )
2023-09-05 17:34:28 +00:00
- [details on the fingerprinted user agent ](#details-on-the-fingerprinted-user-agent )
2023-08-31 14:19:14 +00:00
- [licence ](#licence )
2023-06-02 19:39:25 +00:00
```text
2023-06-03 10:26:40 +00:00
$ surplus 9R3J+R9 Singapore
2023-09-04 14:37:05 +00:00
surplus version 2.1.0
2023-09-02 20:05:37 +00:00
Thomson Plaza
301 Upper Thomson Road
Sin Ming, Bishan
574408
Central, Singapore
2023-06-02 19:39:25 +00:00
```
2023-08-31 14:19:14 +00:00
## installation
2023-06-02 19:39:25 +00:00
2023-09-03 14:23:40 +00:00
> [!IMPORTANT]
2023-09-01 06:17:47 +00:00
> python 3.11 or later is required due to a bug in earlier versions.
> [(python/cpython#88089)](https://github.com/python/cpython/issues/88089)
2023-09-03 14:23:40 +00:00
for most, you can install surplus built from the latest stable release:
2023-06-02 19:39:25 +00:00
```text
2023-09-03 15:49:41 +00:00
pip install https://github.com/markjoshwel/surplus/releases/latest/download/surplus-latest-py3-none-any.whl
2023-06-02 19:39:25 +00:00
```
2023-09-03 14:23:40 +00:00
or directly from the repository using pip:
```text
pip install git+https://github.com/markjoshwel/surplus.git@main
```
surplus is also a public domain dedicated [single python file ](surplus/surplus.py ), so
feel free to grab that and embed it into your own program as you see fit.
see [licence ](#licence ) for licensing information.
2023-09-02 20:05:37 +00:00
## usage
### command-line usage
2023-06-03 08:31:02 +00:00
2023-06-02 19:39:25 +00:00
```text
2023-09-05 17:34:28 +00:00
usage: surplus [-h] [-d] [-v] [-c {pluscode,localcode,latlong,sharetext}]
[-u USER_AGENT]
2023-09-01 16:43:59 +00:00
[query ...]
2023-06-02 19:39:25 +00:00
2023-09-01 13:38:03 +00:00
Google Maps Plus Code to iOS Shortcuts-like shareable text
2023-06-02 19:39:25 +00:00
positional arguments:
2023-09-05 17:34:28 +00:00
query full-length Plus Code (6PH58QMF+FX), shortened
Plus Code/'local code' (8QMF+FX Singapore),
latlong (1.3336875, 103.7749375), string query
(e.g., 'Wisma Atria'), or '-' to read from stdin
2023-06-02 19:39:25 +00:00
options:
2023-09-01 16:43:59 +00:00
-h, --help show this help message and exit
-d, --debug prints lat, long and reverser response dict to
stderr
-v, --version prints version information to stderr and exits
2023-09-05 17:34:28 +00:00
-c {pluscode,localcode,latlong,sharetext}, --convert-to {pluscode,localcode,latlong,sharetext}
2023-09-02 06:32:26 +00:00
converts query a specific output type, defaults
2023-09-02 20:05:37 +00:00
to 'sharetext'
2023-09-05 17:34:28 +00:00
-u USER_AGENT, --user-agent USER_AGENT
user agent string to use for geocoding service,
defaults to fingerprinted user agent string
2023-06-02 19:39:25 +00:00
```
2023-09-02 20:05:37 +00:00
### example api usage
here are a few examples to get you quickly started using surplus in your own program:
1. let surplus do the heavy lifiting
```python
>>> from surplus import surplus, Behaviour
>>> result = surplus("Ngee Ann Polytechnic, Singapore", Behaviour())
>>> result.get()
'Ngee Ann Polytechnic\n535 Clementi Road\nBukit Timah\n599489\nNorthwest, Singapore'
```
2. handle queries seperately
2023-09-03 14:23:40 +00:00
```python
>>> import surplus
>>> behaviour = surplus.Behaviour("6PH58R3M+F8")
>>> query = surplus.parse_query(behaviour)
>>> result = surplus.surplus(query.get(), behaviour)
>>> result.get()
'MacRitchie Nature Trail\nCentral Water Catchment\n574325\nCentral, Singapore'
```
2023-09-02 20:05:37 +00:00
3. start from a Query object
```python
>>> import surplus
>>> localcode = surplus.LocalCodeQuery(code="8R3M+F8", locality="Singapore")
>>> pluscode_str = localcode.to_full_plus_code(geocoder=surplus.default_geocoder).get()
>>> pluscode = surplus.PlusCodeQuery(pluscode_str)
>>> result = surplus.surplus(pluscode, surplus.Behaviour())
>>> result.get()
'Wisma Atria\n435 Orchard Road\n238877\nCentral, Singapore'
```
notes:
2023-09-03 14:23:40 +00:00
- you can change what surplus does by passing in a custom [`Behaviour` ](#class-behaviour )
object
2023-09-02 20:05:37 +00:00
2023-09-03 14:23:40 +00:00
- most surplus functions return a [`Result` ](#class-result ) object. while you can
call [`.get()` ](#resultget ) to obtain the proper return value, this is dangerous and
might raise an exception
2023-09-02 20:05:37 +00:00
see the [api reference ](#api-reference ) for more information.
2023-08-31 14:19:14 +00:00
## developer's guide
prerequisites:
2023-09-01 06:17:47 +00:00
- [Python >=3.11 ](https://www.python.org/ )
2023-08-31 14:19:14 +00:00
- [Poetry ](https://python-poetry.org/ )
alternatively, use [devbox ](https://get.jetpack.io/devbox ) for a hermetic development environment powered by [Nix ](https://nixos.org/ ).
```text
devbox shell # skip this if you aren't using devbox
poetry install
poetry shell
```
2023-09-02 20:05:37 +00:00
for information on surplus's exposed api, see the [api reference ](#api-reference ).
2023-08-31 14:19:14 +00:00
## contributor's guide
1. fork the repository and branch off from the `future` branch
2. make and commit your changes!
3. pull in any changes from `future` , and resolve any conflicts, if any
4. **commit your copyright waiver** (_see below_)
5. submit a pull request (_or mail in a diff_)
when contributing your first changes, please include an empty commit for a copyright
waiver using the following message (replace 'Your Name' with your name or nickname):
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
Your Name Copyright Waiver
I dedicate any and all copyright interest in this software to the
public domain. I make this dedication for the benefit of the public at
large and to the detriment of my heirs and successors. I intend this
dedication to be an overt act of relinquishment in perpetuity of all
present and future rights to this software under copyright law.
```
the command to create an empty commit is `git commit --allow-empty`
### reporting incorrect output
2023-09-03 14:23:40 +00:00
> [!NOTE]
2023-08-31 14:19:14 +00:00
> this section is independent from the rest of the contributing section.
different output from the iOS Shortcuts app is expected, however incorrect output is not.
#### the reporting process
open an issue in the
[repositories issue tracker ](https://github.com/markjoshwel/surplus/issues/new ),
2023-08-31 14:41:25 +00:00
and do the following:
2023-08-31 14:19:14 +00:00
2023-08-31 14:41:25 +00:00
1. ensure that your issue is not an error of incorrect data returned by your reverser
function, which by default is OpenStreetMap Nominatim.
(_don't know what the above means? then you are using the default reverser._)
2023-08-31 14:19:14 +00:00
2023-09-05 17:34:28 +00:00
also look at the ['what counts as "incorrect"' ](#what-counts-as-incorrect ) section
2023-08-31 14:41:25 +00:00
before moving on.
2023-08-31 14:19:14 +00:00
2023-08-31 14:41:25 +00:00
2. include the erroneous query.
(_the Plus Code/local code/latlong coord/query string you passed into surplus_)
2023-08-31 14:19:14 +00:00
3. include output from the teminal with the
2023-08-31 14:41:25 +00:00
[`--debug` flag ](#command-line-usage ) passed to the surplus CLI or with
2023-08-31 14:19:14 +00:00
`debug=True` set in function calls.
2023-09-03 14:23:40 +00:00
> [!NOTE]
2023-08-31 14:41:25 +00:00
> if you are using the surplus API and have passed custom stdout and stderr parameters
> to redirect output, include that instead.
2023-08-31 14:19:14 +00:00
2023-08-31 14:41:25 +00:00
4. how it should look like instead, with reasoning if the error is not obvious. (e.g.,
missing details)
2023-08-31 14:19:14 +00:00
for reference, see how the following issues were written:
- [issue #4: "Incorrect format: repeated lines" ](https://github.com/markjoshwel/surplus/issues/4 )
- [issue #6: "Incorrect format: missing details" ](https://github.com/markjoshwel/surplus/issues/6 )
- [issue #12: "Incorrect format: State before county" ](https://github.com/markjoshwel/surplus/issues/12 )
#### what counts as "incorrect"
2023-08-31 14:41:25 +00:00
- **example** (correct)
2023-08-31 14:19:14 +00:00
- iOS Shortcuts Output
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
Plaza Singapura
68 Orchard Rd
238839
Singapore
```
- surplus Output
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
Plaza Singapura
68 Orchard Road
Museum
238839
Central, Singapore
```
this _should not_ be reported as incorrect, as the only difference between the two is
that surplus displays more information.
other examples that _should not_ be reported are:
- name of place is incorrect/different
this may be due to incorrect data from the geolocator function, which is OpenStreetMap Nominatim by default.
2023-09-03 14:23:40 +00:00
in the case of Nominatim, it means that the data on OpenStreetMap is incorrect.
2023-08-31 14:19:14 +00:00
(_if so, then consider updating OpenStreetMap to help not just you, but other surplus
and OpenStreetMap users!_)
2023-08-31 14:41:25 +00:00
**you should report** when the output does not make logical sense, or something similar
2023-08-31 14:19:14 +00:00
wherein the output of surplus is illogical to read or is not correct in the traditional
sense of a correct address.
2023-08-31 14:41:25 +00:00
see the linked issues in [the reporting process ](#the-reporting-process ) for examples
2023-08-31 14:19:14 +00:00
of incorrect outputs.
## the technical details of surplus's output
2023-09-03 14:23:40 +00:00
> [!NOTE]
2023-09-02 20:05:37 +00:00
> this is a breakdown of surplus's output when converting to shareable text.
2023-09-03 14:23:40 +00:00
> when converting to other output types, output may be different.
2023-09-02 20:05:37 +00:00
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:43:28 +00:00
$ s+ --debug 8QJF+RP Singapore
2023-09-05 17:34:28 +00:00
surplus version 2.1.0, debug mode (latest@future, Tue 05 Sep 2023 23:38:59 +0800)
2023-09-03 14:23:40 +00:00
debug: parse_query: behaviour.query=['8QJF+RP', 'Singapore']
debug: _match_plus_code: portion_plus_code='8QJF+RP', portion_locality='Singapore'
debug: cli: query=Result(value=LocalCodeQuery(code='8QJF+RP', locality='Singapore'), error=None)
2023-09-05 17:34:28 +00:00
debug: cli: latlong_result.get()=Latlong(latitude=1.3320625, longitude=103.7743125)
debug: cli: location={...}
2023-09-03 14:23:40 +00:00
debug: _generate_text: seen_names=['Ngee Ann Polytechnic', 'Clementi Road']
2023-09-02 11:25:54 +00:00
debug: _generate_text_line: [True] -> True -------- 'Ngee Ann Polytechnic'
debug: _generate_text_line: [True] -> True -------- '535'
debug: _generate_text_line: [True] -> True -------- 'Clementi Road'
debug: _generate_text_line: [True, True] -> True -------- 'Bukit Timah'
debug: _generate_text_line: [False, True] -> False filtered 'Singapore'
debug: _generate_text_line: [True] -> True -------- '599489'
debug: _generate_text_line: [True] -> True -------- 'Northwest'
debug: _generate_text_line: [True] -> True -------- 'Singapore'
0 Ngee Ann Polytechnic
1
2
3 535 Clementi Road
4 Bukit Timah
5 599489
6 Northwest, Singapore
Ngee Ann Polytechnic
535 Clementi Road
Bukit Timah
599489
Northwest, Singapore
2023-08-31 14:43:28 +00:00
```
2023-08-31 14:19:14 +00:00
variables
2023-09-02 11:25:54 +00:00
- **variable `behaviour.query` **
2023-08-31 14:19:14 +00:00
2023-09-03 14:23:40 +00:00
the original query string or a list of strings from space-splitting the original query
string passed to [`parse_query()` ](#def-parse_query ) for parsing
2023-08-31 14:19:14 +00:00
```text
$ s+ 77Q4+7X Austin, Texas, USA
--------------------------
query
2023-09-02 11:25:54 +00:00
behaviour.query -> ['77Q4+7X', 'Austin', 'Texas', 'USA']
2023-08-31 14:19:14 +00:00
```
2023-09-03 14:23:40 +00:00
```text
>>> surplus("77Q4+7X Austin, Texas, USA", surplus.Behaviour())
behaviour.query -> '77Q4+7X Austin, Texas, USA'
```
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
- **variables `portion_plus_code` and `portion_locality` **
2023-08-31 14:19:14 +00:00
2023-08-31 14:41:25 +00:00
(_only shown if the query is a local code, not shown on full-length plus codes,
2023-08-31 14:19:14 +00:00
latlong coordinates or string queries_)
represents the plus code and locality portions of a
[shortened plus code ](https://en.wikipedia.org/wiki/Open_Location_Code#Common_usage_and_shortening )
2023-09-03 14:23:40 +00:00
(_referred to as a "local code" in the codebase_) respectively
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
- **variable `query` **
2023-09-03 14:23:40 +00:00
query is a variable of type [`Result` ](#class-result )[`[Query]`](#query)
2023-09-02 11:25:54 +00:00
2023-09-03 14:23:40 +00:00
this variable is displayed to show what query type [`parse_query()` ](#def-parse_query ) has
recognised, and if there were any errors during query parsing
2023-09-02 11:25:54 +00:00
2023-09-05 17:34:28 +00:00
- **expression `latlong_result.get()=` **
2023-08-31 14:19:14 +00:00
(_only shown if the query is a plus code_)
2023-09-03 14:23:40 +00:00
the latitude longitude coordinates derived from the plus code
2023-08-31 14:19:14 +00:00
- **variable `location` **
2023-09-03 14:23:40 +00:00
the response dictionary from the reverser function passed to
[`surplus()` ](#def-surplus )
2023-08-31 14:19:14 +00:00
2023-09-05 17:34:28 +00:00
for more information on the reverser function, see
[`SurplusReverserProtocol` ](#surplusreverserprotocol )
2023-09-02 11:25:54 +00:00
2023-08-31 14:19:14 +00:00
- **variable `seen_names` **
2023-09-05 17:34:28 +00:00
a list of unique important names found in certain Nominatim keys used in final output
2023-09-03 14:23:40 +00:00
lines 0-3
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
- **`_generate_text_line` seen name checks**
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
```text
# filter function boolean list status element
# ============================= ======== ======================
debug: _generate_text_line: [True] -> True -------- 'Ngee Ann Polytechnic'
debug: _generate_text_line: [False, True] -> False filtered 'Singapore'
```
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
a check is done on shareable text line 4 keys (`SHAREABLE_TEXT_LINE_4_KEYS` - general
2023-09-03 14:23:40 +00:00
regional location) to reduce repeated elements found in `seen_names`
2023-06-03 08:31:02 +00:00
2023-09-02 11:25:54 +00:00
reasoning is, if an element on line 4 (general regional location) is the exact same as
2023-09-03 14:23:40 +00:00
a previously seen name, there is no need to include the element
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
- **filter function boolean list**
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
`_generate_text_line` , an internal function defined inside `_generate_text` can be
2023-09-03 14:23:40 +00:00
passed a filter function as a way to filter out certain elements on a line
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
```python
# the filter used in _generate_text, for line 4's seen name checks
filter=lambda ak: [
# everything here should be True if the element is to be kept
ak not in general_global_info,
not any(True if (ak in sn) else False for sn in seen_names),
2023-08-31 14:19:14 +00:00
]
```
2023-09-02 11:25:54 +00:00
`general_global_info` is a list of strings containing elements from line 6. (general
global information)
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
- **status**
2023-08-31 14:19:14 +00:00
2023-09-02 11:25:54 +00:00
what `all(filter(detail))` evaluates to, `filter` being the filter function passed to
`_generate_text_line` and `detail` being the current element
- **element**
the current iteration from iterating through a list of strings containing elements
from line 4. (general regional location)
2023-08-31 14:19:14 +00:00
2023-09-03 14:23:40 +00:00
line breakdown of shareable text output, accompanied by their Nominatim keys:
2023-08-31 14:19:14 +00:00
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
0 name of a place
1 building name
2 highway name
3 block/house/building number, house name, road
4 general regional location
5 postal code
6 general global information
```
0. **name of a place**
(_usually important places or landmarks_)
- examples
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
The University of Queensland
Ngee Ann Polytechnic
Botanic Gardens
```
- nominatim keys
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:41:25 +00:00
emergency, historic, military, natural, landuse, place, railway, man_made,
aerialway, boundary, amenity, aeroway, club, craft, leisure, office, mountain_pass,
2023-08-31 14:19:14 +00:00
shop, tourism, bridge, tunnel, waterway
```
1. **building name**
- examples
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
Novena Square Office Tower A
Visitor Centre
```
- nominatim keys
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
building
```
2. **highway name**
- examples
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
Marina Coastal Expressway
Lornie Highway
```
- nominatim keys
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
highway
```
3. **block/house/building number, house name, road**
- examples
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
535 Clementi Road
Macquarie Street
Braddell Road
```
- nominatim keys
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
house_number, house_name, road
```
4. **general regional location**
- examples
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
St Lucia, Greater Brisbane
The Drag, Austin
Toa Payoh Crest
```
- nominatim keys
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:41:25 +00:00
residential, neighbourhood, allotments, quarter, city_district, district, borough,
2023-08-31 14:19:14 +00:00
suburb, subdivision, municipality, city, town, village
```
5. **postal code**
- examples
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
310131
78705
4066
```
- nominatim key
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
postcode
```
6. **general global information**
- examples
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
Travis County, Texas, United States
Southeast, Singapore
Queensland, Australia
```
- nominatim keys
2023-09-03 14:23:40 +00:00
```text
2023-08-31 14:19:14 +00:00
region, county, state, state_district, country, continent
```
## api reference
2023-09-02 20:05:37 +00:00
- [constants ](#constants )
- [exception classes ](#exception-classes )
- [types ](#types )
2023-09-03 14:23:40 +00:00
- [`Query` ](#query )
- [`ResultType` ](#resulttype )
2023-09-05 17:34:28 +00:00
- [`SurplusGeocoderProtocol` ](#surplusgeocoderprotocol )
- [`SurplusReverserProtocol` ](#surplusreverserprotocol )
2023-09-02 20:05:37 +00:00
- [`class Behaviour` ](#class-behaviour )
2023-09-05 17:34:28 +00:00
- [`class SurplusDefaultGeocoding` ](#class-surplusdefaultgeocoding )
- [`SurplusDefaultGeocoding.update_geocoding_functions()` ](#surplusdefaultgeocodingupdate_geocoding_functions )
- [`SurplusDefaultGeocoding.geocoder()` ](#surplusdefaultgeocodinggeocoder )
- [`SurplusDefaultGeocoding.reverser()` ](#surplusdefaultgeocodingreverser )
2023-09-02 20:05:37 +00:00
- [`class ConversionResultTypeEnum` ](#class-conversionresulttypeenum )
- [`class Result` ](#class-result )
- [`Result.__bool__()` ](#result__bool__ )
- [`Result.cry()` ](#resultcry )
- [`Result.get()` ](#resultget )
- [`class Latlong` ](#class-latlong )
2023-09-03 14:23:40 +00:00
- [`Latlong.__str__()` ](#latlong__str__ )
2023-09-02 20:05:37 +00:00
- [`class PlusCodeQuery` ](#class-pluscodequery )
2023-09-03 14:23:40 +00:00
- [`PlusCodeQuery.to_lat_long_coord()` ](#pluscodequeryto_lat_long_coord )
- [`PlusCodeQuery.__str__()` ](#pluscodequery__str__ )
2023-09-02 20:05:37 +00:00
- [`class LocalCodeQuery` ](#class-localcodequery )
2023-09-03 14:23:40 +00:00
- [`LocalCodeQuery.to_full_plus_code()` ](#localcodequeryto_full_plus_code )
- [`LocalCodeQuery.to_lat_long_coord()` ](#localcodequeryto_lat_long_coord )
- [`LocalCodeQuery.__str__()` ](#localcodequery__str__ )
2023-09-02 20:05:37 +00:00
- [`class LatlongQuery` ](#class-latlongquery )
2023-09-03 14:23:40 +00:00
- [`LatlongQuery.to_lat_long_coord()` ](#latlongqueryto_lat_long_coord )
- [`LatlongQuery.__str__()` ](#latlongquery__str__ )
2023-09-02 20:05:37 +00:00
- [`class StringQuery` ](#class-stringquery )
2023-09-03 14:23:40 +00:00
- [`StringQuery.to_lat_long_coord()` ](#stringqueryto_lat_long_coord )
- [`StringQuery.__str__()` ](#stringquery__str__ )
2023-09-02 20:05:37 +00:00
- [`def surplus()` ](#def-surplus )
- [`def parse_query()` ](#def-parse_query )
2023-09-05 17:34:28 +00:00
- [`def generate_fingerprinted_user_agent` ](#def-generate_fingerprinted_user_agent )
- [details on the fingerprinted user agent ](#details-on-the-fingerprinted-user-agent )
2023-09-02 20:05:37 +00:00
### constants
- `VERSION: tuple[int, int, int]`
a tuple of integers representing the version of surplus, in the format
`[major, minor, patch]`
2023-09-05 17:34:28 +00:00
- `VERSION_SUFFIX: typing.Final[str]`
`BUILD_BRANCH: typing.Final[str]`
`BUILD_COMMIT: typing.Final[str]`
`BUILD_DATETIME: typing.Final[datetime]`
2023-09-04 16:49:45 +00:00
string and a [datetime.datetime ](https://docs.python.org/3/library/datetime.html ) object
containing version and build information, set by [releaser.py ](releaser.py )
2023-09-05 17:34:28 +00:00
- `CONNECTION_MAX_RETRIES: int = 9`
`CONNECTION_WAIT_SECONDS: int = 10`
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
defines if and how many times to retry a connection, alongside how many seconds to wait
in between tries, for Nominatim
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
> [!NOTE]
> this constant only affects the default surplus Nominatim geocoding functions. custom
> functions do not read from this, unless deliberately programmed to do so
- `SHAREABLE_TEXT_LINE_0_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_1_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_2_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_3_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_4_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_5_KEYS: typing.Final[tuple[str, ...]]`
`SHAREABLE_TEXT_LINE_6_KEYS: typing.Final[tuple[str, ...]]`
a tuple of strings containing Nominatim keys used in shareable text line 0-6
- `SHAREABLE_TEXT_NAMES: typing.Final[tuple[str, ...]]`
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
a tuple of strings containing Nominatim keys used in shareable text line 0-2 and
2023-09-02 20:05:37 +00:00
special keys in line 3
2023-09-05 17:34:28 +00:00
- `EMPTY_LATLONG: typing.Final[Latlong]`
2023-09-02 20:05:37 +00:00
a constant for an empty latlong coordinate, with latitude and longitude set to 0.0
### exception classes
- `class SurplusException(Exception)`
base skeleton exception for handling and typing surplus exception classes
- `class NoSuitableLocationError(SurplusException)`
- `class IncompletePlusCodeError(SurplusException)`
- `class PlusCodeNotFoundError(SurplusException)`
- `class LatlongParseError(SurplusException)`
- `class EmptyQueryError(SurplusException)`
- `class UnavailableFeatureError(SurplusException)`
### types
#### `Query`
```python
Query: typing.TypeAlias = PlusCodeQuery | LocalCodeQuery | LatlongQuery | StringQuery
```
[type alias ](https://docs.python.org/3/library/typing.html#type-aliases ) representing
either a
[`PlusCodeQuery` ](#class-pluscodequery ),
[`LocalCodeQuery` ](#class-localcodequery ),
[`LatlongQuery` ](#class-latlongquery ) or
[`StringQuery` ](#class-stringquery )
#### `ResultType`
```python
ResultType = TypeVar("ResultType")
```
[generic type ](https://docs.python.org/3/library/typing.html#generics ) used by
[`Result` ](#class-result )
2023-09-05 17:34:28 +00:00
#### `SurplusGeocoderProtocol`
[typing_extensions.Protocol ](https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols )
class for documentation and static type checking of surplus geocoder functions
- **signature and conforming function signature**
```python
class SurplusGeocoderProtocol(Protocol):
def __call__ (self, place: str) -> Latlong:
...
```
functions that conform to this protocol should have the following signature:
```python
def example(place: str) -> Latlong: ...
```
- **information on conforming functions**
function takes in a location name as a string, and returns a [Latlong ](#class-latlong )
function can and should be be
[`functools.lru_cache()`-wrapped ](https://docs.python.org/3/library/functools.html#functools.lru_cache )
if the geocoding service asks for caching
exceptions are handled by the caller
#### `SurplusReverserProtocol`
[typing_extensions.Protocol ](https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols )
class for documentation and static type checking of surplus reverser functions
- **signature and conforming function signature**
```python
class SurplusReverserProtocol(Protocol):
def __call__ (self, latlong: Latlong, level: int = 18) -> dict[str, Any]:
...
```
functions that conform to this protocol should have the following signature:
```python
def example(self, latlong: Latlong, level: int = 18) -> dict[str, Any]: ...
```
- **information on conforming functions**
function takes in a [Latlong ](#class-latlong ) object and return a dictionary with [`SHAREABLE_TEXT_LINE_*_KEYS` ](#constants ) keys at the dictionaries' top-level.
keys are used to access address information.
function should also take in an int representing the level of detail for the returned
address, 0-18 (country-level to building), inclusive. should default to 18.
keys for latitude, longitude and an iso3166-2 (or closest equivalent) should also be
included at the dictionaries top level as the keys `latitude` , `longitude` and
`ISO3166-2` (non-case sensitive, or at least something starting with `ISO3166` )
respectively.
```python
{
'ISO3166-2-lvl6': 'SG-03',
'amenity': 'Ngee Ann Polytechnic',
...
'country': 'Singapore',
'latitude': 1.33318835,
'longitude': 103.77461234638255,
'postcode': '599489',
'raw': {...},
}
```
function can and should be
[`functools.lru_cache()`-wrapped ](https://docs.python.org/3/library/functools.html#functools.lru_cache )
if the geocoding service asks for caching
see the [playground notebook ](/playground.ipynb ) in repository root for detailed
sample output
exceptions are handled by the caller
2023-09-02 20:05:37 +00:00
### `class Behaviour`
2023-09-03 14:23:40 +00:00
[`typing.NamedTuple` ](https://docs.python.org/3/library/typing.html#typing.NamedTuple )
2023-09-02 20:05:37 +00:00
representing how surplus operations should behave
attributes
- `query: str | list[str] = ""`
2023-09-03 14:23:40 +00:00
original user-passed query string or a list of strings from splitting user-passed query
2023-09-02 20:05:37 +00:00
string by spaces
2023-09-05 17:34:28 +00:00
- `geocoder: SurplusGeocoderProtocol = default_geocoding.geocoder`
name string to location function, see
[`SurplusGeocoderProtocol` ](#surplusgeocoderprotocol ) for more information
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
- `reverser: SurplusReverserProtocol = default_geocoding.reverser`
Latlong object to address information dictionary function, see
[`SurplusReverserProtocol` ](#surplusreverserprotocol ) for more information
2023-09-02 20:05:37 +00:00
- `stderr: typing.TextIO = sys.stderr`
[TextIO-like object ](https://docs.python.org/3/library/io.html#text-i-o )
representing a writeable file.
defaults to [`sys.stderr` ](https://docs.python.org/3/library/sys.html#sys.stderr ).
- `stdout: typing.TextIO = sys.stdout`
[TextIO-like object ](https://docs.python.org/3/library/io.html#text-i-o )
representing a writeable file.
defaults to [`sys.stdout` ](https://docs.python.org/3/library/sys.html#sys.stdout ).
- `debug: bool = False`
whether to print debug information to stderr
- `version_header: bool = False`
whether to print version information and exit
- `convert_to_type: ConversionResultTypeEnum = ConversionResultTypeEnum.SHAREABLE_TEXT`
what type to convert the query to
2023-09-05 17:34:28 +00:00
### `class SurplusDefaultGeocoding`
> [!IMPORTANT]
> this has replaced the now deprecated default geocoding functions, `default_geocoder()`
> and `default_reverser()`, in surplus 2.1.0 and later.
see [SurplusGeocoderProtocol ](#surplusgeocoderprotocol ) and
[SurplusReverserProtocol ](#surplusreverserprotocol ) for more information how to
implement a compliant custom geocoder functions.
[`dataclasses.dataclass` ](https://docs.python.org/3/library/dataclasses.html ) providing
the default geocoding functionality for surplus, via
[OpenStreetMap Nominatim ](https://nominatim.openstreetmap.org/ )
attributes
- `user_agent: str = default_fingerprint`
pass in a custom user agent here, else it will be the default
[fingerprinted user agent ](#details-on-the-fingerprinted-user-agent )
example usage
```python
from surplus import surplus, Behaviour, SurplusDefaultGeocoding
geocoding = SurplusDefaultGeocoding("custom user agent")
geocoding.update_geocoding_functions() # not necessary but recommended
behaviour = Behaviour(
...,
geocoder=geocoding.geocoder,
reverser=geocoding.reverser
)
result = surplus("query", behaviour=behaviour)
...
```
methods
- [`def update_geocoding_functions(self) -> None: ...` ](#surplusdefaultgeocodingupdate_geocoding_functions )
- [`def geocoder(self, place: str) -> Latlong: ...` ](#surplusdefaultgeocodinggeocoder )
- [`def reverser(self, latlong: Latlong, level: int = 18) -> dict[str, Any]: ...`](#surplusdefaultgeocodingreverser)
#### `SurplusDefaultGeocoding.update_geocoding_functions()`
re-initialise the geocoding functions with the current user agent, also generate a new
user agent if not set properly
it is recommended to call this before using surplus as by default the geocoding functions
are uninitialised
- signature
```python
def update_geocoding_functions(self) -> None: ...
```
#### `SurplusDefaultGeocoding.geocoder()`
> [!WARNING]
> this function is primarily given to be passed into a [`Behaviour`](#class-behaviour)
> object, and is not meant to be called directly.
default geocoder for surplus
see [SurplusGeocoderProtocol ](#surplusgeocoderprotocol ) for more information on surplus
geocoder functions
#### `SurplusDefaultGeocoding.reverser()`
> [!WARNING]
> this function is primarily given to be passed into a [`Behaviour`](#class-behaviour)
> object, and is not meant to be called directly.
default reverser for surplus
see [SurplusReverserProtocol ](#surplusreverserprotocol ) for more information on surplus
reverser functions
2023-09-02 20:05:37 +00:00
### `class ConversionResultTypeEnum`
[enum.Enum ](https://docs.python.org/3/library/enum.html )
representing what the result type of conversion should be
values
- `PLUS_CODE: str = "pluscode"`
- `LOCAL_CODE: str = "localcode"`
- `LATLONG: str = "latlong"`
- `SHAREABLE_TEXT: str = "sharetext"`
### `class Result`
2023-09-03 14:23:40 +00:00
[`typing.NamedTuple` ](https://docs.python.org/3/library/typing.html#typing.NamedTuple )
2023-09-02 20:05:37 +00:00
representing the result for safe value retrieval
attributes
- `value: ResultType`
value to return or fallback value if erroneous
- `error: BaseException | None = None`
exception if any
example usage
```python
# do something
def some_operation(path) -> Result[str]:
try:
file = open(path)
contents = file.read()
except Exception as exc:
# must pass a default value
2023-09-03 14:23:40 +00:00
return Result[str]("", error=exc)
2023-09-02 20:05:37 +00:00
else:
2023-09-03 14:23:40 +00:00
return Result[str](contents)
2023-09-02 20:05:37 +00:00
# call function and handle result
result = some_operation("some_file.txt")
if not result: # check if the result is erroneous
# .cry() raises the exception
# (or returns it as a string error message using string=True)
result.cry()
...
else:
# .get() raises exception or returns value,
# but since we checked for errors this is safe
print(result.get())
```
methods
- [`def __bool__(self) -> bool: ...` ](#result__bool__ )
2023-09-03 14:23:40 +00:00
- [`def cry(self, string: bool = False) -> str: ...` ](#resultcry )
- [`def get(self) -> ResultType: ...` ](#resultget )
2023-09-02 20:05:37 +00:00
#### `Result.__bool__()`
method that returns `True` if `self.error` is not `None`
- signature
```python
def __bool__ (self) -> bool: ...
```
- returns `bool`
#### `Result.cry()`
method that raises `self.error` if is an instance of `BaseException` , returns
`self.error` if is an instance of str, or returns an empty string if `self.error` is None
- signature
```python
def cry(self, string: bool = False) -> str: ...
```
- arguments
- `string: bool = False`
if `self.error` is an Exception, returns it as a string error message
- returns `str`
#### `Result.get()`
method that returns `self.value` if Result is non-erroneous else raises error
- signature
```python
def get(self) -> ResultType: ...
```
- returns `self.value`
### `class Latlong`
2023-09-03 14:23:40 +00:00
[`typing.NamedTuple` ](https://docs.python.org/3/library/typing.html#typing.NamedTuple )
2023-09-02 20:05:37 +00:00
representing a latitude-longitude coordinate pair
attributes
- `latitude: float`
- `longitude: float`
methods
- [`def __str__(self) -> str: ...` ](#latlong__str__ )
#### `Latlong.__str__()`
method that returns a comma-and-space-seperated string of `self.latitude` and
`self.longitude`
- signature
```python
def __str__ (self) -> str: ...
```
- returns `str`
### `class PlusCodeQuery`
2023-09-03 14:23:40 +00:00
[`typing.NamedTuple` ](https://docs.python.org/3/library/typing.html#typing.NamedTuple )
2023-09-02 20:05:37 +00:00
representing a full-length Plus Code (e.g., 6PH58QMF+FX)
attributes
- `code: str`
methods
- [`def to_lat_long_coord(self, ...) -> Result[Latlong]: ...`](#pluscodequeryto_lat_long_coord)
- [`def __str__(self) -> str: ...` ](#pluscodequery__str__ )
#### `PlusCodeQuery.to_lat_long_coord()`
- signature
```python
2023-09-05 17:34:28 +00:00
def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
2023-09-02 20:05:37 +00:00
...
```
- arguments
2023-09-05 17:34:28 +00:00
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol ](#surplusgeocoderprotocol ) for more information
2023-09-02 20:05:37 +00:00
- returns [`Result` ](#class-result )[`[Latlong]`](#class-latlong)
#### `PlusCodeQuery.__str__()`
method that returns string representation of query
- signature
```python
def __str__ (self) -> str: ...
```
- returns `str`
### `class LocalCodeQuery`
2023-09-03 14:23:40 +00:00
[`typing.NamedTuple` ](https://docs.python.org/3/library/typing.html#typing.NamedTuple )
2023-09-02 20:05:37 +00:00
representing a
[shortened Plus Code ](https://en.wikipedia.org/wiki/Open_Location_Code#Common_usage_and_shortening )
with locality, referred to by surplus as a "local code"
attributes
- `code: str`
Plus Code portion of local code, e.g., "8QMF+FX"
- `locality: str`
remaining string of local code, e.g., "Singapore"
methods
- [`def to_full_plus_code(self, ...) -> Result[str]: ...`](#localcodequeryto_full_plus_code)
- [`def to_lat_long_coord(self, ...) -> Result[Latlong]: ...`](#localcodequeryto_lat_long_coord)
- [`def __str__(self) -> str: ...` ](#localcodequery__str__ )
#### `LocalCodeQuery.to_full_plus_code()`
exclusive method that returns a full-length Plus Code as a string
- signature
```python
2023-09-05 17:34:28 +00:00
def to_full_plus_code(self, geocoder: SurplusGeocoderProtocol) -> Result[str]:
2023-09-02 20:05:37 +00:00
...
```
- arguments
2023-09-05 17:34:28 +00:00
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol ](#surplusgeocoderprotocol ) for more information
2023-09-02 20:05:37 +00:00
- returns [`Result` ](#class-result )`[str]`
#### `LocalCodeQuery.to_lat_long_coord()`
method that returns a latitude-longitude coordinate pair
- signature
```python
2023-09-05 17:34:28 +00:00
def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
2023-09-02 20:05:37 +00:00
...
```
- arguments
2023-09-05 17:34:28 +00:00
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol ](#surplusgeocoderprotocol ) for more information
2023-09-02 20:05:37 +00:00
- returns [`Result` ](#class-result )[`[Latlong]`](#class-latlong)
#### `LocalCodeQuery.__str__()`
method that returns string representation of query
- signature
```python
def __str__ (self) -> str: ...
```
- returns `str`
### `class LatlongQuery`
2023-09-03 14:23:40 +00:00
[`typing.NamedTuple` ](https://docs.python.org/3/library/typing.html#typing.NamedTuple )
2023-09-02 20:05:37 +00:00
representing a latitude-longitude coordinate pair
attributes
- `latlong: Latlong`
methods
- [`def to_lat_long_coord(self, ...) -> Result[Latlong]: ...`](#latlongqueryto_lat_long_coord)
- [`def __str__(self) -> str: ...` ](#latlongquery__str__ )
#### `LatlongQuery.to_lat_long_coord()`
method that returns a latitude-longitude coordinate pair
- signature
```python
2023-09-05 17:34:28 +00:00
def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
2023-09-02 20:05:37 +00:00
...
```
- arguments
2023-09-05 17:34:28 +00:00
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol ](#surplusgeocoderprotocol ) for more information
2023-09-02 20:05:37 +00:00
- returns [`Result` ](#class-result )[`[Latlong]`](#class-latlong)
#### `LatlongQuery.__str__()`
method that returns string representation of query
- signature
```python
def __str__ (self) -> str: ...
```
- returns `str`
### `class StringQuery`
2023-09-03 14:23:40 +00:00
[`typing.NamedTuple` ](https://docs.python.org/3/library/typing.html#typing.NamedTuple )
2023-09-02 20:05:37 +00:00
representing a pure string query
attributes
- `query: str`
methods
2023-09-03 14:23:40 +00:00
- [`def to_lat_long_coord(self, ...) -> Result[Latlong]: ...`](#stringqueryto_lat_long_coord)
- [`def __str__(self) -> str: ...` ](#stringquery__str__ )
2023-09-02 20:05:37 +00:00
2023-09-03 14:23:40 +00:00
#### `StringQuery.to_lat_long_coord()`
2023-09-02 20:05:37 +00:00
method that returns a latitude-longitude coordinate pair
- signature
```python
2023-09-05 17:34:28 +00:00
def to_lat_long_coord(self, geocoder: SurplusGeocoderProtocol) -> Result[Latlong]:
2023-09-02 20:05:37 +00:00
...
```
- arguments
2023-09-05 17:34:28 +00:00
- `geocoder: SurplusGeocoderProtocol`
name string to location function, see
[SurplusGeocoderProtocol ](#surplusgeocoderprotocol ) for more information
2023-09-02 20:05:37 +00:00
- returns [`Result` ](#class-result )[`[Latlong]`](#class-latlong)
2023-09-03 14:23:40 +00:00
#### `StringQuery.__str__()`
2023-09-02 20:05:37 +00:00
method that returns string representation of query
- signature
```python
def __str__ (self) -> str: ...
```
- returns `str`
### `def surplus()`
query to shareable text conversion function
- signature
```python
def surplus(query: Query | str, behaviour: Behaviour) -> Result[str]: ..
```
- arguments
- `query: str | Query`
[query object ](#query ) to convert or string to attempt to query for then convert
- `behaviour: Behaviour`
[surplus behaviour namedtuple ](#class-behaviour )
- returns [`Result` ](#class-result )`[str]`
### `def parse_query()`
function that parses a query string into a query object
- signature
```python
def parse_query(behaviour: Behaviour) -> Result[Query]: ...
```
- arguments
- `behaviour: Behaviour`
[surplus behaviour namedtuple ](#class-behaviour )
- returns [`Result` ](#class-result )[`[Query]`](#query)
2023-09-05 17:34:28 +00:00
### `def generate_fingerprinted_user_agent()`
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
function that attempts to return a unique user agent string.
2023-09-02 20:05:37 +00:00
- signature
2023-09-05 17:34:28 +00:00
```python
def generate_fingerprinted_user_agent() -> Result[str]:
```
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
- returns [`Result[str]`](#class-result)
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
this result will always have a valid value as erroneous results will have a
resulting value of `'surplus/<version> (generic-user)'`
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
valid results will have a value of `'surplus/<version> (<fingerprin hasht>)'` , where
the fingerprint hash is a 12 character hexadecimal string
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
#### details on the fingerprinted user agent
2023-09-02 20:05:37 +00:00
2023-09-05 17:34:28 +00:00
**why do this in the first place?**
if too many people use surplus at the same time,
Nominatim will start to think it's just one person being greedy. so to prevent this,
surplus will try to generate a unique user agent string for each user through
fingerprinting.
at the time of writing, the pre-hashed fingerprint string is as follows:
```python
unique_info: str = f"{version}-{system_info}-{hostname}-{mac_address}"
```
it contains the following, in order, alongside an example:
1. `version` - the surplus version alongside a suffix, if any
```text
2.1.0-local
```
2. `system_info` - generic machine and operating system information
```text
Linux-6.5.0-locietta-WSL2-xanmod1-x86_64-with-glibc2.35
```
3. `hostname` - your computer's hostname
```text
mark
```
4. `mac_address` - your computer's mac address
```text
A9:36:3C:98:79:33
```
after hashing, this string becomes a 12 character hexadecimal string, as shown below:
```text
surplus/2.1.0-local (1fdbfa0b0cfb)
^^^^^^^^^^
this is the hashed result of unique_info
```
2023-09-06 12:01:50 +00:00
if at any time the retrieval of any of these four elements fail, surplus will just give
2023-09-05 17:34:28 +00:00
up and default to `'surplus/<version> (generic-user)'` .
if any of this seems weird to you, that's fine. pass in a custom user agent flag to
surplus with `-u` or `--user-agent` to override the default user agent, or override the
default user agent in your own code by passing in a custom user agent string to
[`Behaviour` ](#class-behaviour ).
```text
$ surplus --user_agent "a-shiny-custom-and-unique-user-agent" 77Q4+7X Austin, Texas, USA
...
```
```python
>>> from surplus import surplus, Behaviour
>>> surplus(..., Behaviour(user_agent="a-shiny-custom-and-unique-user-agent"))
...
```
2023-06-03 08:31:02 +00:00
2023-08-31 14:19:14 +00:00
## licence
2023-06-02 19:39:25 +00:00
2023-08-31 14:19:14 +00:00
surplus is free and unencumbered software released into the public domain. for more
information, please refer to the [UNLICENCE ](/UNLICENCE ), < https: // unlicense . org > , or the
python module docstring.
2023-09-03 14:23:40 +00:00
however, direct dependencies of surplus are licensed under different, but still permissive
and open-source licences.
- [geopy ](https://pypi.org/project/geopy/ ):
Python Geocoding Toolbox
2023-09-05 17:34:28 +00:00
MIT Licence
2023-09-03 14:23:40 +00:00
- [geographiclib ](https://pypi.org/project/geographiclib/ ):
The geodesic routines from GeographicLib
2023-09-05 17:34:28 +00:00
MIT Licence
2023-09-03 14:23:40 +00:00
- [pluscodes ](https://pypi.org/project/pluscodes/ ):
Compute Plus Codes (Open Location Codes)
Apache 2.0